year3 by lev17755

VIEWS: 11 PAGES: 98

									Massachusetts Mathematics and Science Partnership Title IIB
Annual State-level Evaluation Report
Reporting Period:
February 2, 2004 through August 31, 2006


Prepared for the Massachusetts Department of Education

March 2007
Annual Report of the MMSP State-level Evaluation                                                                                                  Contents




  Contents



                 Executive Summary...................................................................................................................3
                 Program Description .................................................................................................................9
                 Evaluation Plan and Activities ................................................................................................11
                                       State-level Evaluation .......................................................................................11
                                       Local Evaluation and Related Technical Assistance ..........................................11
                 MMSP Participant Background Data.......................................................................................13
                                       State-level Participant Background Data ...........................................................13
                                       Partnership-level Participant Background Data .................................................20
                 Progress Toward Meeting MMSP Goals .................................................................................31
                                       Goal I. ..............................................................................................................31
                                       Goal II. .............................................................................................................37
                                       Goal III. ............................................................................................................44
                                       Goal IV. ............................................................................................................47
                                       Goal V. .............................................................................................................53
                 Discussion and Recommendations ........................................................................................54
                 Appendix A: Timeline for State-level Evaluation and TA Activities ......................................57
                 Appendix B: Minimum Expectations for Evaluation ..............................................................59
                 Appendix C: Participant Background Survey – Year 1 ..........................................................62
                 Appendix D: Participant Background Survey – Year 2 ..........................................................66
                 Appendix E: Participant Background Survey – Year 3 ..........................................................73
                 Appendix F: Results of the Participant Background Survey .................................................81
                 Appendix G: High Need District Eligibility .............................................................................92
                 Appendix H: Subject Matter Competency Demonstration Options .......................................95
                 Appendix I: Highly Qualified Status of Unique Participants .................................................96
                 Appendix J: Criteria that Account for Gain in Highly Qualified Status .................................97




      UMass Donahue Institute
      Research and Evaluation Group                                                                                                                           I
Annual Report of the MMSP State-level Evaluation                                                                                                       Tables




   Tables Index

Table 1: Partnership Budgets by Program Year .......................................................................................9
Table 2: Teaching Areas..............................................................................................................................14
Table 3: Degrees Held and Pursued in Relation to Subjects Taught and Licensure.......................15
Table 4: Types of Schools of Unique Participants .................................................................................16
Table 5: High Need Status of Unique Participants from Public Schools ...........................................17
Table 6: High Need Districts by Partnership ...........................................................................................17
Table 7: Reasons for Participation in the Course – All Seats ..............................................................19
Table 8: EduTron Participant Background Information ........................................................................21
Table 9: Harvard Participant Background Information .........................................................................22
Table 10: Lesley University Participant Background Information ......................................................23
Table 11: MCLA Science Participant Background Information ...........................................................24
Table 12: Salem State College Participant Background Information .................................................25
Table 13: Springfield/Holyoke Participant Background Information ..................................................26
Table 14: Wareham Participant Background Information ....................................................................27
Table 15: WPI Participant Background Information ..............................................................................28
Table 16: MCLA Math Participant Background Information .................................................................29
Table 17: University of Massachusetts Amherst Participant Background Information..................30
Table 18: Repeat Participants by Partnership ........................................................................................31
Table 19: Enrollment and Attrition Information as Reported by Each MMSP Partnership.............32
Table 20: Highly Qualified Status of Unique Participants ....................................................................38
Table 21: Number of Unique Participants Gaining Highly Qualified Status, by Partnership .........39
Table 22: MTEL Tests Taken by MMSP Participants – Total to Date ..................................................40
Table 23: HOUSSE Plan Status of Unique Public School Teachers ...................................................41
Table 24a: MMSP Science and Technology Teaching Areas—Regular Education .........................41
Table 24b: MMSP Science and Technology Teaching Areas—Special Education ..........................42
Table 25a: MMSP Mathematics Teacher Levels —Regular Education ...............................................42
Table 25b: MMSP Mathematics Teacher Levels —Special Education ...............................................43
Table 26: Mean Percentage Scores for Pre-course & Post-course Tests, Including Gains in
Mean Scores ...................................................................................................................................................47




        UMass Donahue Institute
        Research and Evaluation Group                                                                                                                          II
Annual Report of the MMSP State-level Evaluation                                                                   Executive Summary




    Executive Summary

The purpose of the Massachusetts Mathematics and Science Partnership (MMSP) Program is to provide high
quality professional development for teachers in Massachusetts in the content areas of mathematics, science, and
technology/engineering (MSTE). This multi-year project is funded through Title IIB of the No Child Left Behind
(NCLB) Act. The intention of the MMSP funding is to increase the number of highly qualified teachers in the
specified content areas in the Commonwealth, particularly in high need districts, with an ultimate outcome of
increased student achievement. The funding is administered by the Massachusetts Department of Education
(MADOE).

The Program began in February 2004, and has had three funding periods, defined as follows:
    Year 1: February 2, 2004 through August 31, 2004

          Year 2: September 1, 2004 through August 31, 2005.

          Year 3: September 1, 2005 through August 31, 2006.

The partnerships who received initial funding in Year 1 are referred to as Cohort 1; those who received initial
funding in Year 2 are referred to as Cohort 2. Cohort 1 consisted of eight partnerships, with six of the eight
partnerships offering mathematics professional development and two offering science professional development.
Cohort 2 consisted of two partnerships, both offering mathematics professional development.

The University of Massachusetts Donahue Institute was contracted to coordinate state-level collection of outcome
data and to provide evaluation-related technical assistance to the partnerships.

The MADOE established the following five goals for the partnerships to achieve through MMSP funding. Data
supporting each goal were collected from February 2, 2004 through August 31, 2006, for both Cohort 1 and
Cohort 21.

Goal I.      Develop and implement an effective and sustained course of study for current teachers of grades
             4-8 mathematics and/or grades 4-12 science and technology/engineering.

          A total of ten partnerships were funded across the Commonwealth. Eight were organized around
           mathematical content, and two were organized around science content. Of the ten MMSP partnerships, all
           delivered courses.

          In total, 85 MMSP courses were offered by the end of Year 3 of MMSP funding. Of these 85 courses, 76
           were mathematics courses, and eight were science courses.


1
  Most data are reported as Year 1, Year 2, Year 3, and cumulative totals. The data from Year 1 were collected from February 2, 2004
through August 31, 2004. The data collected in Year 2 were collected from September 1, 2004, to August 31, 2005. The data collected in
Year 3 were collected from September 1, 2005, to August 31, 2006. There are also cumulative totals reported that include all three years’
years’ data. The data reported are largely descriptive in nature. Any participant data that could be aggregated at the state level were
collected from the Participant Background Survey, which each participant completed on the last day of each course. Participant data, unless
noted, are reported by unique individuals regardless of the number of courses taken by each individual. Partnership data regarding
integration of MMSP courses into education and/or arts and sciences at the partnerships’ institutes of higher education was obtained
through a series of items that appeared in the context of the Annual Report Addendum that was completed by partnerships in November
2006.


         UMass Donahue Institute
         Research and Evaluation Group                                                                                                    3
Annual Report of the MMSP State-level Evaluation                                                Executive Summary


         In total, 909 unique participants participated in MMSP courses by the end of Year 3. Of all 909 unique
          participants, 97% came from public schools (including 3% from public charter schools), 2% came from
          non-public schools, and less than 1% did not indicate their school type. Approximately 60% of
          participants from public schools were from high need districts.

         There were 354 participants who took more than one MMSP course by the end of Year 3. Of the 354
          participants who took more than one course, approximately 60% were from high need districts.

Goal II. Increase the number of teachers currently employed in the partnership school districts who are
         licensed in the areas they teach and/or have completed their High Objective Uniform State
         Standard of Evaluation (HOUSSE) plans.

Public school teachers must meet the federal definition of highly qualified to comply with the federal No Child
Left Behind legislation. Goal II addresses the spirit of the federal No Child Left Behind legislation regarding
teacher licensure, professional development, and competency in subject area taught.

         Licensure in Mathematics and Science Content Areas for Middle and High School Teachers

             67% of regular education middle and high school MMSP teachers who taught mathematics were
              licensed in mathematics.

             18% of special education middle and high school MMSP teachers who taught mathematics were
              licensed in mathematics.

             35% of regular education middle and high school MMSP teachers who taught a science or technology
              content area were licensed in the area in which they taught.

             3% of special education middle and high school teachers who taught a science or technology content
              area were licensed in the area in which they taught.

         Highly Qualified Status

             By the end of Year 3, of the participants who had entered MMSP as not highly qualified, 111 had
              attained highly qualified status. The breakdown of how highly qualified status was attained is as
              follows:

                  Sixteen passed the appropriate Massachusetts Tests for Educator Licensure (MTEL).
                  Five obtained a degree in content areas taught.
                  Three earned a teaching license.
                  Three obtained undergraduate degree equivalents in content areas taught.
                  Fourteen simultaneously met two or more criteria.
                  Seventy of the 332 who indicated they had HOUSSE plans completed a sufficient number of
                   Professional Development Points (PDPs) on HOUSSE plans.




        UMass Donahue Institute
        Research and Evaluation Group                                                                              4
Annual Report of the MMSP State-level Evaluation                                                                  Executive Summary


         HOUSSE Plan Information2

             Of 883 public school participants, 38% reported having a HOUSSE Plan, 32% reported not having a
              HOUSSE Plan, and 27% were unsure.

             By the end of Year 3, of the 332 public school participants who indicated they had HOUSSE Plans,
              253 (76%) reported having 48 or more PDP hours, 32 (10%) reported having fewer than 48 PDP
              hours, and 47 (14%) did not respond.

Goal III. Increase the number of highly qualified teachers in mathematics, science, and/or
          technology/engineering by integrating the courses of study into schools of education and/or arts
          and sciences at institutes of higher education.

For Year 3, partnerships were asked to describe the activities of their partnerships during the Year 3 funding
period that spoke to the “institutionalization” of their courses, the extent to which their courses have been
integrated into activities of their higher education partners. The extent and type of integration varied across
partnerships. Following are summaries for partnerships that indicated that additional integration was occurring in
Year 3:

         In Year 3, all MMSP courses that EduTron created were offered through the Center for Professional
          Studies at Fitchburg State College. All courses were offered to in-service teachers for graduate credit. In
          addition, the partnership united Mathematics faculty and Education faculty at Fitchburg State College in a
          fundamental way: All parties are now working together to improve teacher preparation in mathematics
          through more stringent requirements in math courses and by improving mathematics offerings. The
          education department is in the process of hosting a math education summit to articulate the problems and
          present potential solutions that are arrived at through the joint efforts of math and education faculty.

         In addition to continuing with the programs that were created in prior MMSP years, Lesley University has
          begun developing an online mathematics program that will offer a master’s degree in elementary or
          middle school mathematics education. Through needs that have arisen through the classroom
          implementation component of their participation in MMSP, Lesley also has developed a relationship with
          Cisco Corporation who will be providing the online platform for the online mathematics program.

         Massachusetts College of Liberal Arts (MCLA) Science now offers two new variable credit science for
          educators courses for in-service teachers.

         Massachusetts College of Liberal Arts (MCLA) Math now offers one new variable credit math for
          educators courses for in-service teachers.

         Salem State College continues to offer courses developed through MMSP as part of a master’s level
          teaching program in middle school mathematics. All courses developed by Salem State College through
          MMSP can be applied towards earning a degree through that program.

         The three science courses offered by the Springfield partnership through the MMSP program in Year 3
          were offered for graduate credit through University of Massachusetts (UMass) Continuing Education. A
          goal of the partnership is to have these courses institutionalized in the UMass School of Continuing
          Education through the School of Education.

2
  Less than half of all course participants who took more than one course were consistent across surveys in their responses to the survey
item asking them if they had a HOUSSE Plan, indicating that many were confused about the issue. Consequently, data regarding HOUSSE
plans should be viewed with caution.


        UMass Donahue Institute
        Research and Evaluation Group                                                                                                   5
Annual Report of the MMSP State-level Evaluation                                                   Executive Summary



         All courses developed through the UMass Amherst partnership can be taken for graduate level credits that
          can be transferred to a variety of graduate level programs at UMass. In addition, the two courses are being
          reviewed by the School of Education Academic matters Committee for permanent course approval.

         The one new course created by WPI in Year 3 was embedded into the master’s level graduate program in
          mathematics education offered by WPI’s Mathematics Department.

Goal IV. Increase the number of MSTE teachers currently employed in the partner school districts who
         participate in content-based professional development activities and substantially increase their
         content knowledge in order to be able to teach effectively the state learning standards.

         Of the 85 courses offered across all partnerships by the end of Year 3, pre- to post-course test score gains
          were statistically significant in 79 (93%), indicating the courses had a positive impact on participants’
          content knowledge.

         Content assessments for three courses offered through one partnership were administered to both
          treatment (i.e., MMSP) and comparison groups. For each course, the treatment group post-course scores
          were statistically significantly higher than those of the comparison group, although in one case the
          treatment group pre-course scores also were statistically significantly higher than those of the comparison
          group.

         78% of the participants reported taking MMSP courses to increase knowledge in the content offered.

Goal V. Improve student academic achievement as measured by Massachusetts Comprehensive
        Assessment System (MCAS) mathematics and science and technology/engineering assessments
        and other assessments.

         Partnerships reported student data directly to MADOE and to the United States Education Department
          under the federal reporting system.

         The Donahue Institute state-level data collection effort for Title IIB did not include student outcome data;
          however, more than 89,000 students were taught annually by MMSP participants.

Data Quality
While in many instances for this project the data are sound, there are some areas for which this was not the case.
The Donahue Institute became aware that problems existed with some of the data when 1) surveys received from
participants taking multiple courses revealed inconsistencies in reporting across surveys for individual
participants, especially for items used in determining highly qualified status, and 2) surveys indicated that many
participants were confused about whether or not they had a HOUSSE plan.

The Participant Background Survey was modified after Year 1 to improve the likelihood of obtaining good
quality data, but similar problems with the data arose again in Year 2. The survey was modified again in Year 3,
but some problems still existed, and totals to date will be affected by problems from Year 1 and Year 2; therefore,
information regarding the highly qualified status of teachers must be viewed with caution.

Reaching Targeted Participants
The partnerships exceeded the MADOE goal of having at least 50% of all participants come from high need
districts. Half of the ten partnerships providing courses by the end of Year 3 met that goal as individual
partnerships.


        UMass Donahue Institute
        Research and Evaluation Group                                                                                    6
Annual Report of the MMSP State-level Evaluation                                                    Executive Summary


One obstacle that interfered with efforts to raise the high need district participation involved one partnership, in
particular, that had committed to one cohort going through multiple courses over the life of the project. For this
partnership, it was not possible to change their participants mid-program.
     MADOE should continue to remind partnerships that enrollment must include a high percentage of
        participants from high need districts.

         MADOE should encourage partnerships to recruit additional participants from their high need partners
          and/or identify additional high need partners, even if they meet the minimum participant rate of 50%.

         MADOE has already begun revisiting the procedure by which districts are identified as high need
          districts. MADOE also should consider developing criteria to identify individual schools as high need.

Local Evaluation Designs
It was determined early in the initial meetings with partnerships that implementing a rigorous evaluation design –
using randomized-controlled trials or strong comparison groups – would be difficult, if not impossible. The
limitations that arose were limited resources, the lack of sufficient participant numbers to form reasonable
treatment and control groups, and insufficient time to recruit participants into a control group before the start of
the program. At the end of Year 3, only one partnership had implemented a weak quasi-experimental evaluation
design. The remaining partnerships implemented basic evaluation models utilizing pre-post comparisons. To
address the difficulties in creating rigorous evaluation designs, the Request for Proposals for MMSP programs for
2006-2007 included information on how partnerships might obtain additional support through MADOE and the
Donahue Institute for this purpose.

         MADOE and the Donahue Institute should continue to work closely with partnerships to provide
          technical support and resources to conduct rigorous evaluations in the continuation process.

Collecting Student Data
Perhaps the most pervasively problematic issue regarding evaluation of MMSP projects has been difficulty
accessing student data. This has been both a logistical and legal challenge for evaluators. Logistically, many
school districts do not routinely code their student data by teacher; thus, it is difficult to obtain data aggregated at
the participant level. Partnerships have trouble enlisting the necessary cooperation from participating districts
with few teachers enrolled in MMSP courses. Finally, Family Educational Rights and Privacy Act (FERPA)
regulations present a real legal barrier in situations where the evaluation contract is not managed and paid for by
the participating districts – the only relationship that would enable student-level data to be released to the
evaluator. To date, it appears as if only two partnerships, EduTron and Lesley, have made a concerted effort to
obtain student outcome data at the participant (as opposed to school) level.

Because student data was unattainable for most partnerships, they also encountered problems for meeting the
federal reporting requirements regarding student data.

         MADOE should give partnerships guidance on forging agreements with their partner districts to have
          access to student outcome data for participating teachers.

Integrating Courses into Higher Education Institutions
In the second year of the project, efforts were made to define integration and provide partnerships with structured
guidance on reporting progress at integration. In the third year, structured guidance was again provided, and the
data provided evidence that integration is occurring across almost all partnerships and that integration efforts will
continue in the future for many of the partnerships, with at least one partnership (Lesley University) extending
integration efforts beyond the scope of the MMSP grant. While the extent and types of integration that have
occurred varied across partnerships, integration has happened primarily in the following ways that encourage


        UMass Donahue Institute
        Research and Evaluation Group                                                                                      7
Annual Report of the MMSP State-level Evaluation                                           Executive Summary


sustainability beyond the duration of the program: embedding MMSP courses into pre-existing programs,
integrating courses into pre-existing continuing education programs, and creating new degree programs.

         MADOE should encourage partnerships to continue efforts to integrate MMSP courses into degree
          programs of partner higher education institutions.

Steering Committee
The Steering Committee has continued to provide guidance and feedback to the MADOE on issues regarding the
MMSP partnerships, on broader issues of professional development and mathematics, science, technology, and
engineering. The MADOE is fully aware how fortunate it is to have such an engaged and invested Steering
Committee for this initiative. Steering Committee members have made observation visits to most of the
partnerships and have offered programmatic suggestions and recommendations to the department.

         MADOE should continue the role of the Steering Committee providing advice and guidance and make all
          efforts to follow up on committee suggestions and requests.




        UMass Donahue Institute
        Research and Evaluation Group                                                                        8
Annual Report of the MMSP State-level Evaluation                                                Program Description




  Program Description

The purpose of the Massachusetts Mathematics and Science Partnership Program (MMSP) is to provide high
quality professional development for teachers in the content areas of mathematics, science, and technology/
engineering with the goal of increasing the number of highly qualified teachers, particularly in high need districts.
As with most professional development initiatives, the end goal is to improve student achievement levels. This
multi-year project is funded by the U.S. Department of Education as part of the No Child Left Behind (NCLB)
Act Title IIB funding stream. Funding to local partnerships is administered by state education agencies; in
Massachusetts this is the Massachusetts Department of Education (MADOE), which awards funding through a
competitive grant process.

The Program began in February 2004, and has had three funding periods, defined as follows:
    Year 1: February 2, 2004 through August 31, 2004

         Year 2: September 1, 2004 through August 31, 2005.

         Year 3: September 1, 2005 through August 31, 2006.

The partnerships who received initial funding in Year 1 are referred to as Cohort 1; those who received initial
funding in Year 2 are referred to as Cohort 2. Table 1 shows the funding received by each partnership for each
year of the program.



             Table 1: Partnership Budgets by Program Year

             Partnership                                   Year 1            Year 2            Year 3
             EduTron                                       $210,000         $237,000           $323,000
             Harvard Graduate School of Education           $96,743         $188,856           $204,300
             Lesley University                             $220,007         $310,110           $280,609
             MCLA - Science                                 $30,350          $51,912            $50,930
             Salem State College                           $118,395         $209,331           $214,269
             Springfield Public Schools                    $175,000         $151,707           $173,337
             Wareham Public Schools                        $120,930         $162,122           $115,388
             Worcester Polytechnic Institute               $202,322         $203,257           $196,199
             MCLA – Math                                     N/A             $32,864            $78,630
             PV Stemnet                                      N/A             $88,264           $174,151
             TOTAL                                       $1,173,747        $1,635,423        $1,810,813


Cohort 1 consisted of eight partnerships, with six of the eight partnerships offering mathematics professional
development and two offering science professional development. Cohort 2 consisted of two partnerships, both
offering mathematics professional development.




        UMass Donahue Institute
        Research and Evaluation Group                                                                               9
Annual Report of the MMSP State-level Evaluation                                                                  Program Description


This report includes data collected from February 2, 2004 through August 31, 2006, for both Cohort 1 and Cohort
23. Any participant data that could be aggregated at the state level were collected from the Participant Background
Survey, which each participant completed on the last day of each course. Participant data, unless noted, are
reported by unique individuals regardless of the number of courses taken by each individual.

The partnerships are composed of higher education institutions, school districts, and, in some cases, private
organizations involved in providing both pre-service and in-service training to teachers. In order to provide a deep
conceptual understanding of the content, partnerships are encouraged to work with mathematicians and scientists
from the arts and sciences or engineering departments of their higher education partners.

Partnerships are required to offer courses that equal at least 45 hours of course time followed by at least 20 hours
of follow-up support during the school year. Partnerships are encouraged to tailor the model used to deliver the
professional development and follow-up to best fit the objectives of their programs along with their resources,
expertise, and existing infrastructure.

Regardless of the variation of models across the partnerships, the programs are required to meet the following
main goals of the MMSP:

Goal I.     Develop and implement an effective and sustained course of study for current teachers of grades 4-8
            mathematics and/or grades 4-12 science and technology/engineering.

Goal II.    Increase the number of teachers currently employed in the partnership school districts who are licensed
            in the areas they teach and/or have completed their High Objective Uniform State Standard of
            Evaluation (HOUSSE) plans.

Goal III. Increase the number of highly qualified teachers in mathematics, science, and/or technology/
          engineering by integrating the courses of study into schools of education and/or arts and sciences at
          institutes of higher education.

Goal IV. Increase the number of mathematics, science, and technology/engineering teachers currently employed
         in the partner school districts who participate in content-based professional development activities and
         substantially increase their content knowledge in order to be able to teach effectively the state learning
         standards.

Goal V. Improve student academic achievement as measured by Massachusetts Comprehensive Assessment
        System (MCAS) mathematics, science, and technology/engineering assessments and other assessments.

A final component of the program is the formation and on-going role of the MMSP Steering Committee. This
group is composed of mathematicians and scientists, mostly from higher education institutions, as well as public
schools and private organizations. The committee members have a common interest in mathematics, science,
technology, and engineering issues and education. As a group, they provide the MADOE with guidance regarding
programmatic and funding decisions related to the MMSP. This group also provides a mechanism for the
MADOE to receive expert input regarding the mathematical and scientific content being offered by MMSP
partnerships. The Steering Committee is coordinated by the MADOE, and meetings are held periodically
throughout the year.


3
  Most data are reported as Year 1, Year 2, Year 3, and cumulative totals. The data from Year 1 were collected from February 2, 2004
through August 31, 2004. The data collected in Year 2 were collected from September 1, 2004, to August 31, 2005. The data collected in
Year 3 were collected from September 1, 2005, to August 31, 2006. There are also cumulative totals reported that include both years’ data.
The data reported are largely descriptive in nature.


       UMass Donahue Institute
       Research and Evaluation Group                                                                                                    10
Annual Report of the MMSP State-level Evaluation                                         Evaluation Plan and Activities




  Evaluation Plan and Activities

State-level Evaluation

Although not required by the U.S. Department of Education, the Massachusetts Department of Education
contracted with the University of Massachusetts Donahue Institute to conduct a state-level evaluation of the
MMSP. The Donahue Institute’s primary role as state-level evaluator is to coordinate program-wide collection of
outcome data on behalf of the MADOE. Data collection for the state-level evaluation is organized around a basic
logic model for professional development initiatives shown below.




Local Evaluation and Related Technical Assistance

In addition to the state-level data collection, each partnership is required to conduct its own local evaluation. In an
effort to support strong local evaluations, MADOE required that partnerships sub-contract with the Donahue
Institute to provide technical assistance on design and implementation of their local evaluations. The timeline
listing the evaluation activities for this time period is found in Appendix A.

Year 1
Technical assistance activities began at the MMSP Project Kickoff Meeting in February 2004 where the Donahue
Institute presented the following general guidelines for strong local evaluation:
        Include both formative and summative research questions with data collection and analysis plans
         adequate to address those questions
         Utilize experimental or quasi-experimental design where feasible
         Organize data collection around the basic logic model presented above

Following the kickoff meeting, Donahue Institute staff conducted individual meetings with representatives from
each partnership, including evaluators if they had been hired. These meetings provided an opportunity to reinforce


        UMass Donahue Institute
        Research and Evaluation Group                                                                                11
Annual Report of the MMSP State-level Evaluation                                        Evaluation Plan and Activities


the above guidelines for strong local evaluation and present MADOE’s expectations. The latter were documented
in a “minimum expectations” document, a copy of which can be found in Appendix B.

Each of these meetings also included a discussion of whether it might be feasible to adjust the program design to
create an opportunity for an experimental or quasi-experimental evaluation design. It appeared as if the Harvard
partnership might have been able to use a random-assignment design since it was expecting its first course to be
over-enrolled. However, to date that expectation has not come to pass and the program has not been able to
establish a control group for participating teachers. At the end of Year 2, only one program had been able to
implement a weak quasi-experimental design regarding only teacher outcomes. Finally, the Lesley partnership is
hoping to access student assessment data that will allow them to compare outcomes for students of participating
teachers to those for students of teachers who are not participating. It remains to be seen whether those efforts will
be successful and whether student MCAS results will be sensitive enough to provide measurable evidence of
improved student achievement as a result of teachers’ MMSP course participation.

After these initial meetings, the Donahue Institute provided follow-up support for individual partnerships as
necessary.

Year 2
In Year 2, technical assistance was provided for Cohort 1 partnerships as it was needed. The most common
request for assistance was related to completing the end-of-course forms. These questions lessened as staff
became accustomed to filling out the forms.

In the fall of 2004, there was a round of meetings held with key members of each of the Cohort 1 partnerships and
their evaluators to review evaluation plans and provide any assistance necessary. This provided an opportunity to
hear what challenges were faced by evaluators in carrying out their evaluations. The biggest challenge at that time
was the difficulty faced by partnerships in collecting student data. It was clear that unless a relationship had been
forged and the expectation set to collect student achievement data from participating school districts, it would be
very difficult if not impossible to collect these data.

The round of fall meetings also provided an opportunity to introduce the federal reporting tool that had been
released at that time. Although the final draft and timeline had not been set for the reporting period, introducing
the data collection tool gave partnerships the opportunity to collect the necessary data ahead of time.

Technical assistance activities began for the two Cohort 2 partnerships with initial meetings similar to those held
with Cohort 1 partnerships. Because there were only two new partnerships in Cohort 2, a kickoff meeting was not
held. The evaluation models and expectations were presented to the new partnerships at their individual meetings.
The emphasis was on designing solid evaluations to include both formative and summative research questions
with data collection and analysis plans adequate to address those questions. After the initial meetings, follow-up
support was provided for individual partnerships.

In the spring of 2005, the due date and final draft of the federal reporting tool were released. The Donahue
Institute worked with the federal DOE to clarify some of the requirements of the report to fit the Massachusetts
model of professional development. In doing so, it became apparent that partnerships would have many similar
questions. In anticipation of these questions and data requests from partnerships, the Donahue Institute provided a
technical assistance workshop for partnerships to walk through the requirements of the federal reporting tool.
Although partnerships had access to the data required, the Donahue Institute had most of it centralized and easily
accessible. Therefore, the Donahue Institute provided partnerships with the majority of data required for the
federal report.




      UMass Donahue Institute
      Research and Evaluation Group                                                                                   12
Annual Report of the MMSP State-level Evaluation                                 MMSP Participant Background Data




  MMSP Participant Background Data

There are a total of ten MMSP partnerships. The eight partnerships who were initially funded in Year 1, in the
first round of funding, are referred to as Cohort 1. The two partnerships who were initially funded in Year 2 are
referred to as Cohort 2. All MMSP partnerships delivered courses in the three years of MMSP partnership activity
(which, in actuality, spanned over the course of only 30 months). Eight of these partnerships delivered
mathematics content courses, and two delivered science content courses. In total, there were 85 courses delivered.
Of these 85 courses, 76 were mathematics courses, eight were science courses, and one was a technology/
engineering course. This section of the report summarizes data collected from participants in these courses. In
total, there were 909 participants, and 354 of them took two or more courses.

Course participants completed the Participant Background Survey upon completion of each course. See Appendix
C for the survey used in Year 1, see Appendix D for the survey used in Year 2, and see Appendix E for the survey
used in Year 3. The purpose of this survey is to gather data about participants’ professional backgrounds and
qualifications. This information provides a picture of who the participants are, aids in determining whether the
courses are reaching the teachers who most need professional development, and aids in tracking how teacher
qualifications may change during the MMSP funding period. Data from the survey regarding teacher licensure,
possession of and progress towards earning degrees, and status in terms of Massachusetts Tests for Educator
Licensure (MTEL) exams, allows determination of the number of teachers who meet criteria defining highly
qualified status.

 State-level Participant Background Data
By the end of Year 3, 909 unique participants completed the participant background survey on one or more
occasions. The term “unique participant” refers to each individual who participated in the program, regardless of
how many courses he or she took. Data for items from the survey that help to convey participants’ professional
backgrounds and motives for participation are discussed in the remainder of this section. All survey data for this
group may be found in the “Total to Date” column of Appendix F.

Appendix F also contains participant background survey results for each item of the survey for Year 1, Year 2,
and Year 3. Data for each of these years includes only unique participants for that particular year. For example,
Year 3 data includes those participants who completed a course in Year 3 regardless of their participation in either
Year 1 or Year 2, but if a participant completed more than one course in Year 3, his or her data is only counted
once in Year 3.

The responses to the survey questions are presented as frequencies and percentages. Not all percentages total
100% because many items allowed multiple responses and not all of the participants responded to all of the items.

Teaching Experience of Participants
At the time of their last completed survey from an MMSP course, the teaching experience of the 909 unique
participants was as follows: 15% were in their first to third year of teaching, 38% had between 4 and 10 years
experience in education, 27% had between 11 and 20 years of experience, and 18% reported over twenty years of
experience.

Teaching Levels of Participants
The schools in the participating districts were organized into categories of elementary schools (grades K-5), K-8
schools, middle schools (grades 6-8), and high schools (grades 9-12). At the time of their last completed survey


      UMass Donahue Institute
      Research and Evaluation Group                                                                                  13
Annual Report of the MMSP State-level Evaluation                                                 MMSP Participant Background Data


from an MMSP course, 22% of participants were teaching in an elementary or K-8 school, 57% were teaching in
a middle school, and 12% were teaching in a high school.

Content Taught
The distribution of teaching areas of respondents at the time of the survey is shown in Table 2. Because
respondents identified all teaching areas that applied to their positions at the time of the survey, some selected
multiple responses, so frequency totals exceed the number of respondents and percentages exceed 100%. Also,
figures reported in “Total” column may be smaller than figures for any individual year because the total is based
on data from the last survey completed and some repeat participants changed teaching areas over the course of
their participation.

At the time of their last MMSP course, 59% were teaching mathematics, 20% were teaching science, and 21%
were teaching all subjects at the elementary level.



          Table 2: Teaching Areas

                                                            Year 1               Year 2           Year 3              Total
          Teaching Areas
                                                            N = 341              N = 456          N = 464           N = 909
          (Multiple responses permitted)
                                                        n          %*        n          %*       n         %*     n       %*
          Mathematics                                   190        56%       252           55%   288       62%    535     59%
          Any science area                              131        38%       141           31%   107       23%    244     27%
             General Science                              65       19%        71           16%   56        12%    131     14%
             Biology                                      22          6%      24           5%    16         3%     41         5%
             Earth Science                                21          6%      16           4%    14         3%     30         3%
             Chemistry                                    12          4%      14           3%    11         2%     22         2%
             Physics                                      11          3%      16           4%    10         2%     20         2%
          Technology/Engineering                          10          3%         3         1%        4      1%     10         1%
          Computer Science                                   N/A**               2      <1%          3      1%        4   <1%
          Elementary (all subjects)                     103        30%       107           23%   79        17%    189     21%
          Elementary Mathematics                             N/A**            31           7%    26         6%     45         5%
          Other                                           61       18%        25           5%    22         5%     59         6%
          Not Currently Teaching                          13          4%      10           2%    18         4%     32         4%
          TOTAL                                              508                  571                547              1118
          *Percentage of the total unique respondents for relevant year indicating that they currently teach this content area
          **This teaching area not offered as a response option on the Year 1 survey

Position of Participants
At the time of their last completed survey from an MMSP course, 90% of course participants identified
themselves as teachers. Of all respondents, 74% were regular education teachers; 15% were special education or
special education inclusion teachers; 2% were department heads or curriculum coordinators; 1% were principals,
assistant principals, or headmasters; 1% were support specialists; <1% were long-term substitutes; <1% were
superintendents or assistant superintendents; and 5% indicated that they held “other” positions.



      UMass Donahue Institute
      Research and Evaluation Group                                                                                                14
Annual Report of the MMSP State-level Evaluation                                                     MMSP Participant Background Data


Degrees Held
In Year 2, elementary-level teachers were not asked about currently held or pursued degrees, and only secondary-
level mathematics teachers were asked about degrees held or pursued in the area of education. The survey was
subsequently modified to gather degree information from all participants in the third year of this project.

Information on degrees held was derived from the most recently completed survey of each individual. Of the 854
public school teachers who were currently teaching at the time of their last survey, only 688 (81%) responded to
the item asking about degrees held. Data indicated that these 688 reporting teachers held a total of 469 bachelor’s
degrees and 428 master’s degrees. Of the 688 teachers,
        244 (36%) indicated that a bachelor’s degree was their highest degree:
            o 106 were in education
            o 28 were in science
            o 36 were in mathematics
            o 8 were in mathematics education
            o 2 were in science education
            o 81 were in other areas. (Figures total to more than 244 because some indicated that they held
                multiple bachelor’s degrees.)

       428 (62%) indicated that a master’s degree was their highest degree
              o     283 were in education
              o     27 were in science
              o     22 were in mathematics education
              o     14 were in mathematics
              o     6 were in science education
              o     105 were in other areas. (Figures total to more than 428 because some indicated that they held
                    multiple master’s degrees.)

       11 (2%) reported that a Certificate of Advanced Graduate Study (CAGS) was their highest degree: 9 in
         education and 2 in other areas.
       8 (1%) reported that a doctorate was their highest degree, with 2 holding doctorates in education and 6
         holding doctorates in other areas.

Table 3 shows the numbers of individuals – as subgroups of those currently teaching and those licensed – who
currently held degrees in either mathematics or any science, technology, or engineering subject.



       Table 3: Degrees Held and Pursued in Relation to Subjects Taught and Licensure

                                            Currently         Licensed in           Current Degree            Pursuing Degree
                   Area
                                             Teach              Subject              BA       MA         BA        MA       CAGS
    Mathematics*                              570                 316                45       10          2         11        2
    Any STE Subject**                         186                  72                28        5          0          0        1
    *Includes Math, Elementary Math, MS Math, MS Math and Science
    ** Excludes Computer Science because licensure information was not available.



Degrees Currently Pursued
Information on degrees currently being pursued was derived from the most recently completed survey of each
individual. Of the 854 unique public school teachers, 271 (32%) reported pursuing at least one degree. Ten (4%)


      UMass Donahue Institute
      Research and Evaluation Group                                                                                                15
Annual Report of the MMSP State-level Evaluation                                        MMSP Participant Background Data


reported pursuing bachelor’s degrees: 20% in education; 20% in mathematics education; 10% in each of
mathematics, science, science education, and a combination of mathematics education and science education; and
20% in other areas.

Of the 271, 218 (80%) reported a master’s degree as the highest degree pursued: 45% in mathematics education;
29% in education; 12% in mathematics; 2% in science education; 2% in a combination of mathematics education
and science education; 1% in each of the following: science, a combination of mathematics and education, a
combination of mathematics and mathematics education, a combination of education and mathematics education,
a combination of education and science, a combination of mathematics education and another area; and 6% in
other areas. (Percentages total to more than 100% as a result of rounding.)

Of the 271, 35 (13%) reported that a Certificate of Advanced Graduate Study (CAGS) was the highest degree
pursued: 37% in education, 31% in mathematics education, 6% in mathematics, 6% in science, 20% in other
areas.

Of the 271, eight (3%) reported a doctorate degree as the highest degree pursued: 50% in education, and one in
each of the following areas: science, math education, a combination of education and math education, and another
area. (Percentages total to more than 100% as a result of rounding.)

Table 3 shows the numbers of individuals – as subgroups of those currently teaching and those licensed – who
were pursuing degrees in either mathematics or any science, technology, or engineering subject.

Types of Schools of Participants
As shown in Table 4, 97% of MMSP participants worked in a public school setting, and 2% worked in a non-
public school setting.



      Table 4: Types of Schools of Unique Participants

                                                            Year 1          Year 2            Year 3      Total to Date
      School Type
                                                        N       %      N        %         N       %        N       %
      Public School (includes public charter schools)   332     97.4% 448       98.2%    455      98.1%    883    97.1%
      Non-public School                                   8      2.4%   7        1.5%      6       1.3%     20     2.2%
      Other or No Response                                1      0.3%   1        0.2%      3       0.7%      6     0.7%
      TOTAL                                             341    100.1% 456       99.9%    464     100.1%    909   100.0%



High Need Status of Districts of Participants
MMSP partnerships were required to include at least one high need district. Appendix G identifies the criteria for
the high need designation. The high need status of some school districts changed between Year 1 and Year 2. To
classify participants from districts that were high need at one point in time but not high need at another, a process
was used that took into account the high need status of participants’ districts from the beginning of each
partnership’s MMSP involvement with the program. If a district was identified as high need when it first joined
MMSP, that classification continued as long as the district was involved. Specifically, for Year 1, high need status
was determined using only Year 1 eligibility criteria. For Year 2, high need status of Cohort 1 participants was
determined using both Year 1 and Year 2 eligibility criteria, and high need status of Cohort 2 participants was
determined using only Year 2 eligibility criteria. For Year 3, high need status of Cohort 1 participants was
determined using eligibility criteria from Year 1, Year 2, and Year 3. For Year 3, high need status of Cohort 2
participants was determined using eligibility criteria from Year 2 and Year 3.


      UMass Donahue Institute
      Research and Evaluation Group                                                                                       16
Annual Report of the MMSP State-level Evaluation                                                            MMSP Participant Background Data


As of their last course in MMSP, of the 354 individuals who took multiple courses, approximately 60% were from
high need public school districts, approximately 36% were from other public school districts, and approximately
3% either were from private schools or did not provide information on their districts. In addition, approximately
2% were from high need districts for some of the courses they took but not for others 4. Table 5 shows that of the
883 unique participants working in public schools, approximately 60% were employed in high need districts.



       Table 5: High Need Status of Unique Participants from Public Schools

                                                                         Year 1            Year 2               Year 3           Total to Date
       Item
                                                                     N         %          N        %          N         %          N          %
       High Need District                                                202  60.8%        276 61.6%            254 55.8%            530  60.0%
       Non-high Need District                                            129  38.9%        161 35.9%            196 43.1%            337  38.2%
       Other*                                                              1   0.3%         11   2.5%             5   1.1%            16   1.9%
       TOTAL                                                             332 100.0%        448 100.0%           455 100.0%           883 100.1%
       *Includes those who did not identify their districts and public school participants who took multiple courses whose districts were considered
       high need for only some of the courses those participants took.


The MADOE expected that at least 50% of participants in each partnership would come from high need districts,
and further, they set an informal goal that at least 75% of participants for each partnership would come from high
need districts. Over the course of each partnerships involvement in MMSP, five of the ten partnerships had at
least 50% of their participants coming from high need districts, and four of the ten partnerships exceeded the
informal goal of having at least 75% of the participants come from high need districts. Table 6 presents the
number of participants from high need districts organized by each partnership. Note that if columns for Table 6
are summed, the totals will not correspond to relevant data in Table 5 for two reasons: 1) because Table 5 presents
data for unique participants across all partnerships while Table 6 presents data for unique participants only within
partnerships (and some participants took courses through multiple partnerships over all three years) and 2) the
districts of some of those participants who crossed partnerships were not consistently classified as high need
districts (either because of the content of the course or because of the cohort of the partnership offering the
course).



    Table 6: High Need Districts by Partnership

                                                                                      Number of Participants* from High Need Districts
    Partnership                                 High Need Districts                                                         Total to
                                                                                      Year 1      Year 2        Year 3
                                                                                                                              Date
                                                Fitchburg                             37               53                 28               85*
    EduTron                                     Gardner                               14                4                  9               19
                                                Subtotal                              51 (79%)         57 (88%)           37 (84%)        104 (80%)
                                                Boston                                 1                3                 10               13
                                                Boston Renaissance Charter             3                2                  0                5
    Harvard University                          Cambridge                              5                4                  9               17
                                                Fall River                             0                0                  4                4

4
  Teachers who took MMSP math courses when their districts were considered high need for only science were identified as having come
from a “non-high need” district, and teachers who took MMSP science courses when their districts were considered high need for only
math were identified as having come from a “non-high need” district.


       UMass Donahue Institute
       Research and Evaluation Group                                                                                                                   17
Annual Report of the MMSP State-level Evaluation                                     MMSP Participant Background Data



  Table 6: High Need Districts by Partnership

                                                                    Number of Participants* from High Need Districts
  Partnership                          High Need Districts                                                Total to
                                                                    Year 1      Year 2        Year 3
                                                                                                            Date
                                       Lowell                        0           1             0            1
                                       Malden                        0           0            3†            3†
  Harvard University (continued)       New Bedford                   0           1             0            1
                                       Somerville                    0           2             4            6
                                       Somerville Charter School     0           5             0            5
                                       Southbridge                   0           0             1            1
                                       Subtotal                      9 (39%)    18 (33%)      31 (39%)     56 (36%)
                                                                                                †
  Lesley University                    Malden                       21 (21%)    16 (19%)      14 (19%)     21† (20%)
                                       Adams-Cheshire                             6           5             6
                                       Clarksburg                                 1           1             1
                                                                      N/A
  MCLA – Science                       Florida                                    2           1             3
                                       North Adams                                5           3             6
                                       Subtotal                                 14 (100%)    10 (100%)     16 (100%)
                                       Boston                        0            0           1             1
                                       Chelsea                       0            1           1             1
                                       Haverhill Public Schools      1           20          16            29
  Salem State College
                                       Lynn                         32           32          41            69
                                       Salem                        18           16          14            31
                                       Subtotal                     51 (93%)    69 (86%)     73 (79%)     131 (81%)
                                       Holyoke                       6           17          19            30
  Springfield/Holyoke Public Schools
                                       Springfield                  32           31          28            64
                                       Subtotal                     38 (100%)   48 (100%)    47 (96%)      94 (97%)
  Wareham Public Schools               Wareham                      17 (46%)    11 (61%)       N/A         23 (53%)
                                       Abby Kelley Foster Charter    0           2             3            4
                                       Athol-Royalstonβ              1           0             0            0
                                       Berkshire Hills               0           1             0            1
                                       Boston                        0           0             1            1
                                       Brockton                      0           1             0            1
                                       Cambridge                     0           0             1            1
                                       Chicopee                      4           1             2            7
                                       Fall River                    0           1             0            1
                                       Fitchburg                     0           0             1            1*
                                       Lawrence Family Devt. CS      0           0             3            3
  Worcester Polytechnic Institute
                                       Lowell Community CS           0           0             1            1
                                       New Bedford                   2           4             0            4
                                       North Adams                   2           0             0            2
                                       Pittsfield                    0           0             2            2
                                       Ralph C Mahar Charter         2           0             0            2
                                       Seven Hills CS                0           0             2            2
                                       Somerville                    0           1             0            1
                                       Webster                       0           1             0            1
                                       Winchendon                    0           8             6           11
                                       Worcester                     4           7            10           20



      UMass Donahue Institute
      Research and Evaluation Group                                                                                    18
Annual Report of the MMSP State-level Evaluation                                                         MMSP Participant Background Data



   Table 6: High Need Districts by Partnership

                                                                                  Number of Participants* from High Need Districts
   Partnership                                 High Need Districts                                                      Total to
                                                                                  Year 1      Year 2        Year 3
                                                                                                                          Date
   Worcester Polytechnic Institute
                                               Subtotal                           15 (63%)         27 (41%)            32 (43%)        66 (46%)
   (continued)

   MCLA – Math                                 North Adams                         N/A              N/A                1    (9%)        1   (9%)

                                               Chicopee                                             0                  2                2
                                               Greenfield                                           2                  0                2
   University of Massachusetts                 Holyoke                             N/A              5                  5                8
   Amherst                                     Springfield                                          7                  4                9
                                               Westfield                                            2                  2                4
                                               Subtotal                                            16 (64%)            13 (37%)        25 (47%)
    * One participant from Fitchburg took a course through WPI in Year 3 and also took a course through EduTron in a previous year.
    †
      In Year 3, one participant from Malden had taken one course through Harvard and another course through Lesley.
    β
      The Year 1 participant from Athol-Royalston does not appear in the “Total to Date” column because he/she took multiple courses
    and had changed school districts between Year 1 and time of completion of the last survey.

Reasons for Participation in MMSP Courses
For each course taken, respondents identified all of their reasons for participating. Unlike in the earlier portions of
this report where data were presented for unique participants, data for this issue are presented for all course seats,
since responses are uniquely relevant to each distinct course that a participant took. Table 7 presents findings for
all participants for all courses taken for all three years.


  Table 7: Reasons for Participation in the Course – All Seats

   Reasons for Participation                                       Year 1                 Year 2                  Year 3               Total
   (Multiple responses permitted)                              n            %*        n           %*          n            %*      n           %*

   To increase knowledge in content                             355         85%       573          80%        499          72%     1427        78%
   To obtain graduate credit                                    278         66%       456          63%        444          64%     1178        64%
   To earn PDPs for recertification                             161         38%       309          43%        240          35%      710        39%
   To pursue a personal interest                                147         35%       245          34%        207          30%      599        33%
   To get an additional license (certification)                    89       21%       157          22%        103          15%      349        19%
   To prepare for the Massachusetts Test for Educator
                                                                   82       20%       154          21%        118          17%      354        19%
   Licensure (MTEL)
   To earn PDPs for HOUSSE plan requirement                        53       13%       116          16%            90       13%      259        14%
   To follow an administrator’s suggestion                         32        8%           73       10%            47        7%      152         8%
   To obtain a first license (certification)                       15        4%           22        3%            22        3%         59       3%
   Other                                                           25        6%           31        4%            34        5%         90       5%

   TOTAL RESPONSES                                                   1237                  2136                    1804                 5177
    *Percentage of the total number of participants for the year indicating that this was a reason for participation



       UMass Donahue Institute
       Research and Evaluation Group                                                                                                                 19
Annual Report of the MMSP State-level Evaluation                                MMSP Participant Background Data


 Partnership-level Participant Background Data
Presented in Table 8 through Table 17, this section offers an overview of selected participant survey data for each
partnership. The responses to the survey questions are presented as frequencies and percentages. In cases where
not all participants responded to all of the items and in cases where multiple responses were permitted,
percentages presented may not total 100%.
Each of the tables in this section contains information on “Grade of Licensure.” The categories are defined as
follows:
     “Elementary school” refers to the survey options of “PreK-2,” “PreK-3,” “PreK-6,” “PreK-8,” “PreK-9,”
         “1-6,” and “5-6.”
     “Middle school” refers to the survey options of “5-8” and “5-9.”
     “High School” refers to the survey options of “5-12,” “8-12,” and “9-12.”
     “All levels” refers to the survey option of “All levels.”




      UMass Donahue Institute
      Research and Evaluation Group                                                                              20
Annual Report of the MMSP State-level Evaluation                                            MMSP Participant Background Data

MMSP Partnership: EduTron

  Table 8: EduTron Participant Background Information

                                                                              Number of Participants
                                                           Year 1                 Year 2              Year 3        Total to Date
  Total Number of Participants                             65                65                  44                 130
  Participants Who Took Multiple Courses                                                                             49
  Teach Regular Education                                  48        (74%)   44          (68%)   30         (68%)    89     (68%)
  Teach Special Education (sole and inclusion)             12        (18%)   13          (20%)    7         (16%)    23     (18%)
  Teach Elementary (all content areas)                     23        (35%)   33          (51%)   15         (34%)    48     (37%)
  Teach Elementary Math                                        N/A            1          (2%)     1         (2%)      3         (2%)
  Teach Mathematics Above Elementary                       35        (54%)   19          (29%)    1         (2%)     57     (44%)
  Teach Science or Technology                              12        (18%)   13          (20%)   24         (55%)    24     (18%)
  Teach in High Need District                              51        (78%)   57          (88%)   37         (84%)   104     (80%)
  Hold National Certification – Mathematics                    N/A            1          (2%)     1         (2%)      2         (2%)
  Hold National Certification – General Science                N/A            1          (2%)     1         (2%)      2         (2%)
  Hold National Certification – Any Subject               17         (26%)         N/A                N/A                 N/A
  Highly Qualified                        Yes              26        (40%)   34          (52%)   30         (68%)    69     (53%)
                                           No              15        (23%)    3           (5%)    1          (2%)    12      (9%)
                    In some, but not all areas              3         (5%)    3           (5%)    0          (0%)     2      (2%)
                        Cannot be determined               13        (20%)   21          (32%)   10         (23%)    33     (25%)
                Private school/Not applicable               8        (12%)    4           (6%)    3          (6%)    14     (11%)
  Have a HOUSSE Plan                      Yes              24        (37%)   32          (49%)   25         (57%)    56     (43%)
                                           No              26        (40%)    6           (9%)    4          (9%)    28     (22%)
                                    Not Sure               12        (18%)   24          (30%)   13         (30%)    38     (29%)
  Have Passed One or More MTEL Exams                       11        (17%)   20          (31%)   10         (23%)    28     (22%)
  Hold One or More Teaching Licenses                       61        (94%)   65      (100%)      44     (100%)      126     (97%)
                                                                                   Number of Licenses
  Total Number of Licenses Held                                 99                 105                 79                 203
  License Area                              Elementary          35                  48                 30                  83
                              Elementary Mathematics           N/A                   2                  1                   1
                                           Mathematics          17                   7                 10                  27
                                                Science          6                   3                  3                  10
  Grade of Licensure                  Elementary School         43                  66                 39                 106
 (see page 20 for category definitions)   Middle School         24                  20                 21                  48
                                           High School          19                   7                  9                  25
                                             All Levels          9                  12                 10                  22
  Licensed in Areas Taught                          Yes         28                  36                 32                  94
                                                    No          18                  17                  7                  13
                                            Some Areas           8                   2                  1                   9
                                      Unknown and N/A            7                  10                  4                  14




       UMass Donahue Institute
       Research and Evaluation Group                                                                                                   21
 Annual Report of the MMSP State-level Evaluation                                         MMSP Participant Background Data

 MMSP Partnership: Harvard Graduate School of Education (HGSE)

Table 9: Harvard Participant Background Information

                                                                              Number of Participants
                                                           Year 1             Year 2              Year 3         Total to Date
Total Number of Participants                               23                 55                 80                156
Participants Who Took Multiple Courses                                                                              10
Teach Regular Education                                    17         (74%)   45      (82%)      37      (46%)      99      (63%)
Teach Special Education (sole and inclusion)                3         (13%)    5         (9%)    38      (48%)      45      (29%)
Teach Elementary (all content areas)                        1         (4%)     7      (13%)      12      (15%)      20      (13%)
Teach Elementary Math                                           N/A            4         (7%)     6      (8%)       10         (6%)
Teach Mathematics Above Elementary                         18         (78%)   39      (71%)      56      (70%)     108      (69%)
Teach Science or Technology                                 5         (22%)    5         (9%)     4      (5%)       14         (9%)
Teach in High Need District                                 9         (39%)   18      (33%)      31      (39%)      56      (36%)
Hold National Certification – Mathematics                       N/A            5         (9%)     2      (3%)        7         (4%)
Hold National Certification – General Science                   N/A            0         (0%)     0      (0%)        0         (0%)
Hold National Certification – Any Subject                   2         (9%)     N/A                 N/A                N/A
Highly Qualified                          Yes              11         (48%)   29      (53%)      46      (58%)      87      (56%)
                                           No               5         (22%)    8      (14%)      13      (16%)      24      (15%)
                    In some, but not all areas              0          (0%)    1       (2%)       0       (0%)       1       (1%)
                        Cannot be determined                5         (22%)   11      (20%)      15      (19%)      31      (20%)
                Private school/Not applicable               2          (9%)    6      (11%)       6       (8%)      13       (8%)
Have a HOUSSE Plan                        Yes               1          (4%)   15      (27%)      32      (40%)      49      (31%)
                                           No               9         (39%)   22      (39%)      26      (33%)      57      (37%)
                                    Not Sure               12         (52%)   18      (33%)      18      (23%)      45      (29%)
Have Passed One or More MTEL Exams                         10         (43%)   19      (35%)      25      (31%)      53      (34%)
Hold One or More Teaching Licenses                         21         (91%)   52      (95%)      74      (93%)     146      (94%)
                                                                                   Number of Licenses
Total Number of Licenses Held                                   34                  93                131                258
License Area                                 Elementary       8                     30                   39               76
                               Elementary Mathematics       N/A                      1                    1                3
                                            Mathematics      12                     29                   34               75
                Middle School Mathematics/Science             0                      3                    0                2
                                                 Science      4                      1                    0                5
                                             Technology       1                      0                    0                1
Grade of Licensure                     Elementary School      9                     37                   67              114
  (see page 20 for category definitions)   Middle School     18                     35                   31               84
                                            High School       6                     19                   23               47
                                              All Levels      1                      2                   10               13
Licensed in Areas Taught                             Yes     11                     32                   57              107
                                                     No       5                     15                    8               26
                                             Some Areas       1                      2                    1                2
                                       Unknown and N/A        4                      3                   14               21



         UMass Donahue Institute
         Research and Evaluation Group                                                                                                22
 Annual Report of the MMSP State-level Evaluation                                         MMSP Participant Background Data

 MMSP Partnership: Lesley University

Table 10: Lesley University Participant Background Information

                                                                              Number of Participants
                                                           Year 1             Year 2              Year 3           Total to Date
Total Number of Participants                               99                 85                 75                  107
Participants Who Took Multiple Courses                                                                                85
Teach Regular Education                                    75         (75%)   64      (75%)      52      (69%)        76      (71%)
Teach Special Education (sole and inclusion)               14         (14%)   11      (13%)      10      (13%)        13      (12%)
Teach Elementary (all content areas)                       43         (43%)   38      (45%)      25      (33%)        45      (42%)
Teach Elementary Math                                           N/A            7         (8%)    16      (21%)       22       (21%)
Teach Mathematics Above Elementary                         54         (54%)   37      (44%)      42      (56%)        45      (42%)
Teach Science or Technology                                11         (11%)    4         (5%)     3         (4%)       6         (6%)
Teach in High Need District                                21         (21%)   16      (19%)      14      (19%)        21      (20%)
Hold National Certification – Mathematics                       N/A            1         (1%)     1         (1%)       2         (2%)
Hold National Certification – General Science                   N/A            0         (0%)     0         (0%)       1         (1%)
Hold National Certification – Any Subject                   4         (4%)     N/A                 N/A                  N/A
Highly Qualified                          Yes              37         (37%)   45      (53%)      53      (71%)        71      (66%)
                                           No              30         (30%)   19      (22%)       4       (5%)        13      (12%)
                    In some, but not all areas              0          (0%)    3       (4%)       3       (4%)         2       (2%)
                        Cannot be determined               25         (25%)   17      (20%)      11      (15%)        16      (15%)
                Private school/Not applicable               7          (7%)    1       (1%)       4       (5%)         5       (5%)
Have a HOUSSE Plan                        Yes              20         (20%)   32      (38%)      40      (53%)        54      (50%)
                                           No              47         (47%)   32      (38%)      24      (32%)        37      (35%)
                                    Not Sure               27         (27%)   19      (22%)       9      (12%)        13      (12%)
Have Passed One or More MTEL Exams                         27         (27%)   24      (28%)      23      (31%)        32      (30%)
Hold One or More Teaching Licenses                         98         (99%)   82      (96%)      75    (100%)        106      (99%)
                                                                                   Number of Licenses
Total Number of Licenses Held                                   176                140                125                  185
License Area                                 Elementary          74                  6                48                    74
                                Elementary Mathematics          N/A                  0                 2                     2
                                            Mathematics          20                 19                26                    34
                 Middle School Mathematics/Science                0                  2                 1                     1
                                                 Science          6                  2                 3                     3
                                             Technology           0                  0                 2                     0
Grade of Licensure                     Elementary School        103                 79                66                   102
(see page 20 for category definitions)    Middle School          36                 27                31                    42
                                            High School          18                 16                16                    19
                                              All Levels         19                 16                12                    21
Licensed in Areas Taught                             Yes         62                 57                61                    89
                                                     No          26                 20                 2                     4
                                             Some Areas           4                  2                 3                     2
                                       Unknown and N/A            6                  3                 9                    12




          UMass Donahue Institute
          Research and Evaluation Group                                                                                                 23
Annual Report of the MMSP State-level Evaluation                                            MMSP Participant Background Data

MMSP Partnership: Massachusetts College of Liberal Arts (MCLA) Science

     Table 11: MCLA Science Participant Background Information

                                                                                   Number of Participants
                                                                Year 2                  Year 3            Total to Date

     Total Number of Participants                               14                     10                   16
     Participants Who Took Multiple Courses                                                                  9
     Teach Regular Education                                    12         (86%)        9         (90%)     13         (81%)
     Teach Special Education (sole and inclusion)                1          (7%)        1         (10%)      2         (13%)
     Teach Elementary (all content areas)                        1          (7%)        0          (0%)      2         (13%)
     Teach Elementary Math                                       0          (0%)        0          (0%)      0          (0%)
     Teach Mathematics Above Elementary                          7         (50%)        3         (30%)      6         (38%)
     Teach Science or Technology                                 8         (57%)        7         (70%)      9         (56%)
     Teach in High Need District                                14        (100%)       10        (100%)     16        (100%)
     Hold National Certification – Mathematics                   0         (0%)         0          (0%)      0          (0%)
     Hold National Certification – General Science               0         (0%)         0          (0%)      0          (0%)
     Highly Qualified                              Yes           8        (57%)         8         (80%)     11         (69%)
                                                    No           3        (21%)         0          (0%)      1          (6%)
                             In some, but not all areas          1         (7%)         1         (10%)      1          (6%)
                                Cannot be determined             1         (7%)         1         (10%)      2         (13%)
                        Private school/Not applicable            1         (7%)         0          (0%)      1          (6%)
     Have a HOUSSE Plan                            Yes           7        (50%)         8         (80%)      9         (56%)
                                                    No           3        (21%)         1         (10%)      3         (19%)
                                             Not Sure            4        (29%)         1         (10%)      4         (25%)
     Have Passed One or More MTEL Exams                          2         (14%)        3         (30%)      2         (13%)
     Hold One or More Teaching Licenses                         14        (100%)       10        (100%)     16        (100%)
                                                                                    Number of Licenses
     Total Number of Licenses Held                                   24                     14                   28
     License Area                                 Elementary          9                      4                    9
                                    Elementary Mathematics            0                      0                    0
                                                 Mathematics          1                      0                    1
                                                      Science         3                      2                    4
     Grade of Licensure                     Elementary School        11                      4                   10
     (see page 20 for category definitions)     Middle School        11                      4                   11
                                                 High School          2                      4                    5
                                                   All Levels         0                      2                    2
     Licensed in Areas Taught                             Yes         2                      7                    9
                                                          No          8                      2                    5
                                                  Some Areas          2                      1                    1
                                            Unknown and N/A           2                      0                    1




      UMass Donahue Institute
      Research and Evaluation Group                                                                                            24
 Annual Report of the MMSP State-level Evaluation                                         MMSP Participant Background Data

 MMSP Partnership: Salem State College

Table 12: Salem State College Participant Background Information

                                                                              Number of Participants
                                                           Year 1             Year 2              Year 3           Total to Date
Total Number of Participants                               55                 80                 92                  162
Participants Who Took Multiple Courses                                                                                72
Teach Regular Education                                    43         (78%)   67      (84%)      75        (82%)     128      (79%)
Teach Special Education (sole and inclusion)               11         (20%)   10      (13%)       7        (8%)       22      (14%)
Teach Elementary (all content areas)                       15         (27%)   10      (13%)       7        (8%)       20      (12%)
Teach Elementary Math                                           N/A            4         (5%)     2        (2%)        3         (2%)
Teach Mathematics Above Elementary                         39         (71%)   64      (80%)      78        (85%)     129      (80%)
Teach Science or Technology                                19         (35%)   16      (20%)      11        (12%)      28      (17%)
Teach in High Need District                                51         (93%)   69      (86%)      73        (79%)     131      (81%)
Hold National Certification – Mathematics                       N/A            1         (1%)     4        (4%)        5         (3%)
Hold National Certification – General Science                   N/A            0         (0%)     0        (0%)        0         (0%)
Hold National Certification – Any Subject                  15         (27%)    N/A                 N/A                  N/A
Highly Qualified                          Yes              22         (40%)   51      (64%)      74        (80%)     110      (68%)
                                           No              14         (25%)   15      (19%)       8         (9%)      23      (14%)
                    In some, but not all areas              3          (5%)    2       (3%)       6         (7%)       8       (5%)
                        Cannot be determined               16         (29%)   11      (14%)       2         (2%)      18      (11%)
                Private school/Not applicable               0          (0%)    1       (1%)       2         (2%)       3       (2%)
Have a HOUSSE Plan                        Yes              18         (33%)   29      (36%)      31        (34%)      49      (30%)
                                           No              19         (35%)   33      (41%)      36        (39%)      67      (41%)
                                    Not Sure               16         (29%)   17      (21%)      20        (22%)      39      (24%)
Have Passed One or More MTEL Exams                         12         (22%)   26      (33%)      47        (51%)      64      (40%)
Hold One or More Teaching Licenses                         52         (95%)   78      (98%)      87        (95%)     157      (97%)
                                                                                   Number of Licenses
Total Number of Licenses Held                                   88                 130             156                     277
License Area                                 Elementary      35                     39                24                    61
                                Elementary Mathematics      N/A                      0                 3                     4
                                            Mathematics      11                     46                81                   110
                 Middle School Mathematics/Science            2                      0                 4                     6
                                                 Science      4                      8                 9                    17
                                             Technology       1                      0                 0                     0
Grade of Licensure                     Elementary School     50                     54                36                    94
(see page 20 for category definitions)     Middle School     24                     49                56                    95
                                            High School       8                     23                59                    76
                                              All Levels      4                      3                 5                    10
Licensed in Areas Taught                             Yes     23                     43                79                   123
                                                     No      24                     25                 8                    19
                                             Some Areas       4                      7                 3                     9
                                       Unknown and N/A        1                      3                 2                    11




          UMass Donahue Institute
          Research and Evaluation Group                                                                                                 25
 Annual Report of the MMSP State-level Evaluation                                         MMSP Participant Background Data

 MMSP Partnership: Springfield/Holyoke Partnership

Table 13: Springfield/Holyoke Participant Background Information

                                                                              Number of Participants
                                                           Year 1             Year 2              Year 3         Total to Date
Total Number of Participants                               38                 48                 49                 97
Participants Who Took Multiple Courses                                                                              42
Teach Regular Education                                    34         (89%)   32      (67%)      37      (76%)      72      (74%)
Teach Special Education (sole and inclusion)                2          (5%)    5      (10%)       7      (14%)      11      (11%)
Teach Elementary (all content areas)                        0          (0%)    1       (2%)       4      (8%)        5      (5%)
Teach Elementary Math                                           N/A            0       (0%)       0      (0%)        0      (0%)
Teach Mathematics Above Elementary                          4         (11%)    3       (6%)       4      (8%)        8      (8%)
Teach Science or Technology                                36         (95%)   41      (85%)      42      (86%)      82      (85%)
Teach in High Need District                                38     (100%)      48     (100%)      47      (96%)      94      (97%)
Hold National Certification – Mathematics                       N/A            0       (0%)       0      (0%)        0      (0%)
Hold National Certification – General Science                   N/A            0       (0%)       2      (4%)        1      (1%)
Hold National Certification – Any Subject                   7         (18%)    N/A                 N/A                N/A
Highly Qualified                          Yes               8         (21%)   30      (63%)      32      (65%)      50      (52%)
                                           No              11         (29%)    3       (6%)       3       (6%)      13      (13%)
                    In some, but not all areas             12         (32%)    7      (15%)       7      (14%)      16      (17%)
                        Cannot be determined                6         (16%)    7      (15%)       3       (6%)      14      (14%)
                Private school/Not applicable               1          (3%)    1       (2%)       4       (8%)       4       (4%)
Have a HOUSSE Plan                        Yes               3          (8%)   19      (40%)      22      (45%)      33      (34%)
                                           No              20         (53%)    6      (13%)      14      (29%)      26      (27%)
                                    Not Sure               10         (26%)   22      (46%)      13      (27%)      34      (35%)
Have Passed One or More MTEL Exams                         12         (32%)   16      (33%)      20      (41%)      30      (31%)
Hold One or More Teaching Licenses                         34         (89%)   47      (98%)      46      (94%)      91      (94%)
                                                                                   Number of Licenses
Total Number of Licenses Held                                     73                 96               79                 166
License Area                                 Elementary           4                   2                3                   8
                                Elementary Mathematics          N/A                   0                0                   0
                                            Mathematics           1                   1                0                   1
                 Middle School Mathematics/Science                0                   2                4                   3
                                                 Science         51                  75               51                 107
                                             Technology           0                   0                3                   3
Grade of Licensure                     Elementary School          7                   6               11                  21
(see page 20 for category definitions)     Middle School         19                  35               21                  40
                                            High School          43                  51               43                  95
                                              All Levels          4                   1                4                   7
Licensed in Areas Taught                             Yes         10                  11               30                  47
                                                     No           6                  26                8                  24
                                             Some Areas          16                   8                8                  20
                                       Unknown and N/A            2                   2                3                   6




          UMass Donahue Institute
          Research and Evaluation Group                                                                                             26
Annual Report of the MMSP State-level Evaluation                                       MMSP Participant Background Data

MMSP Partnership: Wareham Public Schools

       Table 14: Wareham Participant Background Information

                                                                                Number of Participants
                                                                  Year 1               Year 2          Total to Date
       Total Number of Participants                               37                   18                 43
       Participants Who Took Multiple Courses                                                             12
       Teach Regular Education                                    32         (86%)     13      (72%)      37         (86%)
       Teach Special Education (sole and inclusion)                5         (14%)      4      (22%)       5         (12%)
       Teach Elementary (all content areas)                       20         (54%)      5      (28%)      22         (51%)
       Teach Elementary Math                                           N/A              0      (0%)        1          (2%)
       Teach Mathematics Above Elementary                         19         (51%)     12      (67%)      18         (42%)
       Teach Science or Technology                                 4         (11%)      0      (0%)        2          (5%)
       Teach in High Need District                                17         (46%)     11      (61%)      23         (53%)
       Hold National Certification – Mathematics                       N/A              1      (6%)        1          (2%)
       Hold National Certification – General Science                   N/A              0      (0%)        0          (0%)
       Hold National Certification – Any Subject                   6         (16%)       N/A                   N/A
       Highly Qualified                          Yes              14         (38%)     12      (67%)      22         (51%)
                                                  No              11         (30%)      3      (17%)      11         (26%)
                           In some, but not all areas              0          (0%)      0       (0%)       0          (0%)
                               Cannot be determined               12         (32%)      2      (11%)       9         (21%)
                       Private school/Not applicable               0          (0%)      1       (6%)       1          (2%)
       Have a HOUSSE Plan                        Yes               5         (14%)      6      (33%)      12         (28%)
                                                  No              15         (41%)     11      (61%)      22         (51%)
                                           Not Sure               16         (43%)      1       (6%)       9         (21%)
       Have Passed One or More MTEL Exams                         10         (27%)      6      (33%)      10         (23%)
       Hold One or More Teaching Licenses                         36         (97%)     18   (100%)        43     (100%)
                                                                                 Number of Licenses
       Total Number of Licenses Held                                     57                 34                   68
       License Area                                 Elementary          32                  13                   38
                                       Elementary Mathematics          N/A                   0                    0
                                                   Mathematics           8                   8                   13
                                                        Science          0                   1                    1
       Grade of Licensure                     Elementary School         39                  19                   47
       (see page 20 for category definitions)     Middle School          9                   7                   10
                                                   High School           3                   6                    6
                                                     All Levels          5                   2                    4
       Licensed in Areas Taught                             Yes         25                  13                   41
                                                            No           9                   4                    2
                                                    Some Areas           2                   0                    0
                                              Unknown and N/A            0                   1                    0




      UMass Donahue Institute
      Research and Evaluation Group                                                                                          27
 Annual Report of the MMSP State-level Evaluation                                          MMSP Participant Background Data

 MMSP Partnership: Worcester Polytechnic Institute

Table 15: WPI Participant Background Information

                                                                              Number of Participants
                                                           Year 1             Year 2               Year 3            Total to Date
Total Number of Participants                               24                 66                  74                   145
Participants Who Took Multiple Courses                                                                                  47
Teach Regular Education                                    21         (88%)   55         (83%)    61         (82%)     121         (83%)
Teach Special Education (sole and inclusion)                0         (0%)     8         (12%)     6         (8%)       11         (8%)
Teach Elementary (all content areas)                        1         (4%)     8         (12%)     7         (9%)       16         (11%)
Teach Elementary Math                                           N/A            1         (2%)      2         (3%)        3         (2%)
Teach Mathematics Above Elementary                         21         (88%)   56         (85%)    67         (91%)     121         (83%)
Teach Science or Technology                                 2         (8%)     8         (12%)     9         (12%)      18         (12%)
Teach in High Need District                                15         (63%)   27         (41%)    32         (43%)      66         (46%)
Hold National Certification – Mathematics                       N/A            2         (3%)      5         (7%)        5         (3%)
Hold National Certification – General Science                   N/A            0         (0%)      0         (0%)        1         (1%)
Hold National Certification – Any Subject                   0         (0%)     N/A                  N/A                   N/A
Highly Qualified                          Yes              13         (54%)   42         (64%)    62         (84%)     105         (72%)
                                           No               6         (25%)   11         (17%)     6          (8%)      18         (12%)
                    In some, but not all areas              1          (4%)    4          (6%)     4          (5%)       8          (6%)
                        Cannot be determined                1          (4%)    8         (12%)     0          (0%)      11          (8%)
                Private school/Not applicable               3         (13%)    1          (2%)     2          (3%)       3          (2%)
Have a HOUSSE Plan                        Yes               4         (17%)   24         (36%)    22         (30%)      51         (35%)
                                           No              13         (54%)   22         (33%)    26         (35%)      46         (32%)
                                    Not Sure                5         (21%)   19         (29%)    25         (34%)      44         (30%)
Have Passed One or More MTEL Exams                         13         (54%)   25         (38%)    48         (65%)      73         (50%)
Hold One or More Teaching Licenses                         24     (100%)      63         (95%)    71         (96%)     138         (95%)
                                                                                   Number of Licenses
Total Number of Licenses Held                                    42                114                 128                   244
License Area                                 Elementary          15                 25                 34                     65
                                Elementary Mathematics          N/A                  0                  1                      2
                                            Mathematics          19                 39                 54                     94
                 Middle School Mathematics/Science                0                  4                  4                      7
                                                 Science          1                  8                  4                     14
                                             Technology           0                  0                  1                      1
Grade of Licensure                     Elementary School         18                 29                 40                     76
(see page 20 for category definitions)     Middle School         12                 50                 65                    104
                                            High School           8                 25                 16                     44
                                              All Levels          2                 10                  6                     17
Licensed in Areas Taught                             Yes         17                 39                 62                    117
                                                     No           4                 20                  9                     16
                                             Some Areas           1                  2                  3                      7
                                       Unknown and N/A            2                  2                  0                      5




          UMass Donahue Institute
          Research and Evaluation Group                                                                                                    28
Annual Report of the MMSP State-level Evaluation                                    MMSP Participant Background Data

MMSP Partnership: Massachusetts College of Liberal Arts (MCLA) Math

                 Table 16: MCLA Math Participant Background Information

                                                                              Number of Participants
                                                                                       Year 3
                 Total Number of Participants                                     11
                 Participants Who Took Multiple Courses                            5
                 Teach Regular Education                                           8             (73%)
                 Teach Special Education (sole and inclusion)                      1              (9%)
                 Teach Elementary (all content areas)                              0              (0%)
                 Teach Elementary Math                                             0              (0%)
                 Teach Mathematics Above Elementary                                9             (82%)
                 Teach Science or Technology                                       3             (27%)
                 Teach in High Need District                                       1              (9%)
                 Hold National Certification – Mathematics                         1              (9%)
                 Hold National Certification – General Science                     0              (0%)
                 Highly Qualified                                 Yes              9             (82%)
                                                                   No              0              (0%)
                                            In some, but not all areas             1              (9%)
                                               Cannot be determined                0              (0%)
                                       Private school/Not applicable               1              (9%)
                 Have a HOUSSE Plan                               Yes              9             (82%)
                                                                   No              0              (0%)
                                                            Not Sure               2             (18%)
                 Have Passed One or More MTEL Exams                                4             (36%)

                 Hold One or More Teaching Licenses                               11            (100%)
                                                                               Number of Licenses
                 Total Number of Licenses Held                                           19
                 License Area                                   Elementary                2
                                                    Elementary Mathematics                0
                                                               Mathematics                4
                                             Middle School Math/Science                   1
                                                                   Science                2
                 Grade of Licensure                      Elementary School                3
                 (see page 20 for category definitions)      Middle School               12
                                                               High School                3
                                                                 All Levels               1
                 Licensed in Areas Taught                              Yes                5
                                                                        No                2
                                                               Some Areas                 2*
                                                         Unknown and N/A                  2
                 *One participant was licensed in some areas taught as of the first course taken in Year
                 3, but was not licensed in any areas taught as of the last course in Year 3 because
                 content areas taught changed between the first and last course.



      UMass Donahue Institute
      Research and Evaluation Group                                                                                    29
Annual Report of the MMSP State-level Evaluation                                       MMSP Participant Background Data

MMSP Partnership: University of Massachusetts Amherst

    Table 17: University of Massachusetts Amherst Participant Background Information

                                                                              Number of Participants
                                                                 Year 2                 Year 3          Total to Date
    Total Number of Participants                                25                    35                  53
    Participants Who Took Multiple Courses                                                                34
    Teach Regular Education                                     19        (76%)       26        (74%)     41        (77%)
    Teach Special Education (sole and inclusion)                 5        (20%)        5        (14%)      8        (15%)
    Teach Elementary (all content areas)                         4        (16%)       10        (29%)     13        (25%)
    Teach Elementary Math                                        4        (16%)        1         (3%)      5        (9%)
    Teach Mathematics Above Elementary                          17        (68%)       24        (69%)     35        (66%)
    Teach Science or Technology                                  2        (8%)         2         (6%)      3        (6%)
    Teach in High Need District                                 16        (64%)       13        (37%)     25        (47%)
    Hold National Certification – Mathematics                    0        (0%)         0         (0%)      0        (0%)
    Hold National Certification – General Science                0        (0%)         0         (0%)      0        (0%)
    Highly Qualified                          Yes               15        (60%)       25        (71%)     38        (72%)
                                               No                4        (16%)        1         (3%)      5         (9%)
                        In some, but not all areas               0         (0%)        1         (3%)      0         (0%)
                            Cannot be determined                 4        (16%)        5        (14%)      4         (8%)
                    Private school/Not applicable                2         (8%)        3         (9%)      6        (11%)
    Have a HOUSSE Plan                        Yes               13        (52%)       12        (34%)     20        (38%)
                                               No                7        (28%)       11        (31%)     16        (30%)
                                        Not Sure                 4        (16%)        8        (23%)     13        (25%)
    Have Passed One or More MTEL Exams                           6        (24%)       15        (43%)     19        (36%)

    Hold One or More Teaching Licenses                          21        (84%)       33        (94%)     48        (91%)
                                                                                  Number of Licenses
    Total Number of Licenses Held                                    36                    64                  83
    License Area                                  Elementary         11                    20                  29
                                    Elementary Mathematics            1                     0                   1
                                                Mathematics           7                    19                  21
                     Middle School Mathematics/Science                0                     1                   1
                                                     Science          2                     3                   4
    Grade of Licensure                     Elementary School         16                    28                  39
    (see page 20 for category definitions)     Middle School          8                    16                  18
                                                High School           5                    13                  16
                                                   All Levels         6                     6                   9
    Licensed in Areas Taught                             Yes         11                    27                  38
                                                          No          7                     3                   9
                                                 Some Areas           2                     0                   1
                                           Unknown and N/A            1                     5                   5




       UMass Donahue Institute
       Research and Evaluation Group                                                                                        30
Annual Report of the MMSP State-level Evaluation                                  MMSP Participant Background Data




  Progress Toward Meeting MMSP Goals

MMSP partnerships strive to achieve five major goals. This section presents data collected by the
Donahue Institute at the state-level that illustrates the progress being made toward these goals.
Additional data collected at the partnership level, such as student achievement data, are not collected by
the Donahue Institute and, therefore, are not included in this report.

Goal I.
          Develop and implement an effective and sustained course of study for current teachers of
          grades 4-8 mathematics and/or grades 4-12 science and technology/engineering.

Over the span of Year 1 through Year 3, MMSP partnerships developed and implemented a total of 85
courses. Of those 85 courses, 51 (60%) were unique, and 34 (40%) were repeat offerings. Of the 85
courses, 76 (89%) of the courses offered mathematics content, eight (9%) offered science content, and
one (1%) offered technology/engineering content.

Partnerships were very successful at encouraging and retaining repeat participants. All ten partnerships
offered multiple courses, and all had participants who attended more than one course. In all, 354
participants attended multiple courses. Eleven participants took courses from multiple partnerships.
Table 18 provides details regarding repeat participation.


                    Table 18: Repeat Participants by Partnership

                                              Number of          Total Number           Number
                                               Courses            of Unique*            Taking
                    Partnership
                                              Offered to         Participants to        Multiple
                                                Date                  Date              Courses
                    EduTron                          7                 130                  49
                    Harvard Graduate
                                                     7                 156                  10
                    School of Education
                    Lesley University              18                  107                  85

                    MCLA - Science                   2                  16                    9

                    Salem State College            26                  162                  72

                    Springfield PS                   7                  97                  42

                    Wareham PS                       3                  43                  12
                    Worcester
                                                     6                 145                  47
                    Polytechnic Institute
                    MCLA - Math                      2                  11                    5

                    UMass Amherst                    7                  53                  34
                   * Unique refers to status within each individual partnership. Eleven participants
                   took courses across partnerships.




      UMass Donahue Institute
      Research and Evaluation Group                                                                                  31
     Annual Report of the MMSP State-level Evaluation                                MMSP Participant Background Data

     In total, the partnerships served 909 participants. Course attrition rates were generally low and averaged
     under 4.5%. Of the 85 courses, 42 had an attrition rate of 0%, 22 had an attrition rate ranging between
     2% and 10%, and 19 had an attrition rate of greater than 10%. (It was not possible to determine the
     attrition rate for two of the courses.) Table 19 provides a breakdown, by course, of enrollment and
     attrition rates.


Table 19: Enrollment and Attrition Information as Reported by Each MMSP Partnership

                                                                    Number of          Number of
                                                                    Participants       Participants
Partnership     Year       Course Title                                                                   Attrition Rate
                                                                    Enrolled First     Completed
                                                                    Day                Course
                           Building a Solid Bridge from
                   1                                                       35                 35                 0%
                           Arithmetic to Algebra

                   1       Linear Equations and Their Foundations          41                 40                 2%

                   2       Building a Rock Solid Math Foundation           27                 27                 0%

                           An Odyssey to Algebra, Geometry and             35                 32                 9%
                   2
                           Back
EduTron                    A Further Odyssey to Algebra,
                   2                                                       35                 33                 6%
                           Geometry and Back

                           Perspectives on Elementary                      21                 20                 5%
                   3
                           Mathematics

                           Perspectives on Middle School                   41                 40                 2%
                   3
                           Mathematics

                Subtotal                                                  235                227                3%
                           Conceptually Challenging Topics in
                   1                                                       28                 23                18%
                           Middle School Math (Making the Case )

                   2       Making the Case Ib                              29                 27                 7%

                   2       Making the Case Ic                              24                 24                 0%

                   3       Making the Case I                               31                 17                45%
Harvard
                   3       Making the Case I                               33                 32                 3%
                Summary – Making the Case I                               145                123                15%

                   2       Making the Case II                              15                 15                 0%

                   3       Making the Case I (SPED)                        34                 32                 6%

                Subtotal                                                  194                170               12%

                   1       Math as a Second Language 1                     25                 25                 0%

                   1       Math as a Second Language 2                     36                 34                 6%
Lesley
                   1       Math as a Second Language 3                     41                 40                 2%
University
                Summary – Math as a Second Language                       102                 99                 3%

                   1       Functions and Algebra – cohort 1                19                 19                 0%




             UMass Donahue Institute
             Research and Evaluation Group                                                                                 32
     Annual Report of the MMSP State-level Evaluation                                   MMSP Participant Background Data



Table 19: Enrollment and Attrition Information as Reported by Each MMSP Partnership

                                                                       Number of          Number of
                                                                       Participants       Participants
Partnership      Year       Course Title                                                                     Attrition Rate
                                                                       Enrolled First     Completed
                                                                       Day                Course
                    2       Functions and Algebra – cohort 2                  27                 25                 7%
Lesley
University          2     Functions and Algebra – cohort 3                    41                 41                 0%
                 Summary – Functions and Algebra                              87                 85                 2%
(continued)
                            Trigonometry, Algebra, and Analytic
                    2                                                         34                 34                 0%
                            Geometry – cohort 1

                          Trigonometry, Algebra, and Analytic
                    2                                                         46                 46                 0%
                          Geometry – cohort 2
                 Summary – Trigonometry, Algebra, & Geometry                  80                 80                 0%
                    2       Number Theory – cohort 1                          36                 36                 0%

                    2     Number Theory – cohort 2                            31                 31                 0%
                 Summary – Number Theory                                      67                 67                 0%
                    2       Statistics and Data Analysis – cohort 1           32                 32                 0%

                    2       Statistics and Data Analysis – cohort 2           17                 17                 0%

                    3       Statistics and Data Analysis – cohort 3           23                 23                 0%
                 Summary – Statistics and Data Analysis                       72                 72                 0%
                    3       Measurement and Probability – cohort 1            22                 21                 5%

                    3     Measurement and Probability – cohort 2              47                 47                 0%
                 Summary – Measurement and Probability                        69                 68                 1%
                    3       Calculus I – cohort 1                             27                 27                 0%

                    3     Calculus I – cohort 2                               28                 28                 0%
                 Summary – Calculus I                                         55                 55                 0%
                    3       Calculus II                                        9                  9                 0%

                 Subtotal                                                    541                535                1%

                    2       Conceptual Physical Science                       15                 14                 7%
MCLA
                    3       Conceptual Biology                                11                  9                18%
Science

                 Subtotal                                                     26                 23               12%
                            Data, Probability, & Statistics for ES &
                    1                                                         23                 23                 0%
                            MS Teachers
                            Data, Probability, and Statistics for MS
                    2                                                         13                 11                15%
                            Teachers
Salem State
                          Data, Probability, and Statistics for MS
College             3                                                         19                 19                 0%
                          Teachers
                 Summary – Data, Probability, & Statistics                    55                 53                 4%

                    1       Number Systems for MS Teachers                    13                 13                 0%



              UMass Donahue Institute
              Research and Evaluation Group                                                                                   33
     Annual Report of the MMSP State-level Evaluation                                 MMSP Participant Background Data



Table 19: Enrollment and Attrition Information as Reported by Each MMSP Partnership

                                                                     Number of          Number of
                                                                     Participants       Participants
Partnership      Year      Course Title                                                                    Attrition Rate
                                                                     Enrolled First     Completed
                                                                     Day                Course
                    2      Number Systems for MS Teachers                   30                 30                 0%

                    2      Number Systems for MS Teachers                    7                  7                 0%

                    3      Number Systems for MS Teachers                   17                 17                 0%
                 Summary – Number Systems                                   67                 67                 0%
                          History of Mathematics for MS
                    1                                                        7                  7                 0%
                          Teachers

                    2      History of Math for MS Teachers                  12                 10                17%

                    3      History of Math for MS Teachers                  14                 14                 0%
                 Summary – History of Mathematics                           33                 31                 6%
                          Patterns, Relations & Algebra for MS
                    1                                                       19                 17                11%
                          Teachers
                           Patterns, Relations, and Algebra for MS
                    2                                                       18                 16                11%
                           Teachers

                           Patterns, Relations, and Algebra for MS
                    2                                                       21                 17                19%
                           Teachers

                          Patterns, Relations, and Algebra for MS
                    3                                                       18                 17                 6%
                          Teachers
Salem State
                 Summary – Patterns, Relations, & Algebra                   76                 67                12%
College
                          Geometry & Measurement for MS
(continued)         1                                                       15                 15                 0%
                          Teachers – Spring
                          Geometry & Measurement for MS
                    1                                                       10                  9                10%
                          Teachers – Summer
                           Geometry & Measurement for MS
                    2                                                        6                  6                 0%
                           Teachers

                          Geometry & Measurement for MS
                    3                                                       20                 17                15%
                          Teachers
                 Summary – Geometry & Measurement                           51                 47                 8%
                    2      PreCalculus for MS Teachers                      22                 20                 9%

                    3      PreCalculus for MS Teachers                       9                  8                11%

                    3     PreCalculus for MS Teachers                       21                 21                 0%
                 Summary – Precalculus for MS Teachers                      52                 49                 6%

                    2      Discrete Mathematics for MS Teachers             12                 12                 0%

                    3     Discrete Mathematics for MS Teachers              10                  9                10%
                 Summary – Discrete Mathematics for MS Teachers             22                 21                 5%




              UMass Donahue Institute
              Research and Evaluation Group                                                                                 34
      Annual Report of the MMSP State-level Evaluation                             MMSP Participant Background Data



Table 19: Enrollment and Attrition Information as Reported by Each MMSP Partnership

                                                                  Number of          Number of
                                                                  Participants       Participants
Partnership      Year       Course Title                                                                Attrition Rate
                                                                  Enrolled First     Completed
                                                                  Day                Course
                    2       Calculus for MS Teachers                     16                 14                13%
Salem State
                    3     Calculus for MS Teachers                       14                 13                 7%
College
                 Summary – Calculus for MS Teachers                      30                 27                10%
(continued)
                    3       Linear Systems for MS Teachers               21                 20                 5%

                 Subtotal                                               407                382                6%

                    1       Physics                                      24                 24                 0%


                    1       Chemistry                                    25                 25                 0%


                    2       Life Science                                 29                 29                 0%
Springfield
                    2       Earth Science                                31                 31                 0%
PS
                    3       Math for Science Teachers                    28                 28                 0%

                    3       Technology/Engineering                       25                 25                 0%

                    3       Integrated Science                           34                 34                 0%

                 Subtotal                                               196                196                0%
                            Improving Math Teaching,
                    1                                                    38                 37                 3%
                            Gr. 4-8
                            Topics in Mathematics for Teachers:
                    2                                                    16                 14                13%
                            Improving Math Teaching Part II
Wareham PS
                    2       Developing Teacher Portfolios                 7                  7                 0%


                 Subtotal                                                61                 58                5%

                    1       Geometrical Concepts                     Unknown                29              Unknown

                            Geometry (same as Geometrical
                    3                                                    37                 29                22%
                            Concepts course)
                 Summary – Geometry                                  Unknown                58              Unknown

                    1       Discrete Mathematics                     Unknown                21              Unknown
WPI

                    2       Algebra for Middle School Teachers           68                 57                16%

                    2       Probability and Statistics                   31                 23                26%

                    3       Middle School Math and MCAS Studies          80                 63                21%

                 Subtotal                                            Unknown               222              Unknown

MCLA-Math           3       Problem Solving and Numeration                8                  6                25%



              UMass Donahue Institute
              Research and Evaluation Group                                                                              35
        Annual Report of the MMSP State-level Evaluation                             MMSP Participant Background Data



 Table 19: Enrollment and Attrition Information as Reported by Each MMSP Partnership

                                                                    Number of          Number of
                                                                    Participants       Participants
 Partnership         Year        Course Title                                                             Attrition Rate
                                                                    Enrolled First     Completed
                                                                    Day                Course
 MCLA-Math              3        Patterns, Relations, and Algebra           8                  8                 0%
 (continued)
                    Subtotal                                               16                 14               13%

                        2        Algebra – cohort 1                        17                 16                 6%

                       3     Algebra – cohort 2                            23                 21                 9%
                    Summary - Algebra                                      40                 37                 8%
                        2        Geometry – cohort 1                       20                 20                 0%

 UMass                 3     Geometry – cohort 2                           18                 15                17%
 Amherst            Summary – Geometry                                     38                 35                 8%
                        3        Algebra B (mini course)                    8                  5                38%

                        3        Trigonometry – cohort 1                   13                 13                 0%

                        3        Discrete Mathematics – cohort 1           17                 17                 0%

                    Subtotal                                              116                107                8%
 All
                    TOTAL                                               1792*               1934               5%*
 Partnerships
* Excluding data for WPI partnership, which were not available.




                UMass Donahue Institute
                Research and Evaluation Group                                                                              36
Annual Report of the MMSP State-level Evaluation                        MMSP Participant Background Data

Goal II.
           Increase the number of teachers currently employed in the partnership school districts who
           are licensed in the areas they teach and/or have completed their HOUSSE plans.

Goal II addresses the spirit of the federal NCLB legislation regarding teacher licensure, professional
development, and competency in subject area taught. To comply with the federal NCLB legislation,
public school teachers were required to meet the federal definition of highly qualified by the end of the
2005-2006 school year. One of the expectations of the MMSP is to provide high quality professional
development that would allow teachers to attain federal highly qualified status.

Information regarding the following areas was used to determine participants’ highly qualified status:
licensure, years in education, subject areas taught, HOUSSE plans held, Professional Development
Points (PDPs) held, MTELs passed, degrees held, undergraduate degree equivalents, and advanced or
national certifications.

To be considered highly qualified, a teacher must be licensed and demonstrate subject matter
competency in the areas of teaching. Demonstration of subject matter competency for elementary
teachers was satisfied either by passing the appropriate MTEL (general curriculum MTEL if teaching
multiple core subjects or elementary math MTEL if teaching mainly elementary math) or by having a
HOUSSE plan and having earned a particular number of PDPs. Demonstration of subject matter
competency for middle and secondary school teachers was satisfied by one of the following means:
passing the appropriate MTEL, completing an appropriate undergraduate major or graduate degree,
completing appropriate coursework comparable to an undergraduate major, holding advanced or
national certification in the appropriate subject area, or by having a HOUSSE plan and having earned a
particular number of PDPs. For all teachers with HOUSSE plans, the minimum numbers of PDPs
needed varied in relation to the date of June 30, 2006: Prior to June 30, 2006, the minimum was 48, and
following June 30, 2006, the minimum was 96. Appendix H outlines options available for
demonstrating subject matter competency.

A participant was identified as highly qualified if the criteria for meeting highly qualified status were
met for all subjects that a participant taught. If a participant taught multiple subject areas and only met
the highly qualified criteria for some of the subjects taught, he or she was determined to be “highly
qualified in some, but not all” content areas.

In Year 1, the Participant Background Survey did not adequately capture information about teachers
that could be used to determine highly qualified status. The survey was re-designed for Year 2 to
capture this information. Year 2 survey responses indicate, though, that participants misunderstood the
meaning of HOUSSE plans, and some who completed more than one survey reported inconsistent data
across surveys.

The Participant Background Survey permitted determination of the impact of MMSP courses on highly
qualified status only for 1) those participants who held HOUSSE plans and 2) those participants who
took more than one course (i.e., completed more than one survey). Because surveys were administered
after participants had completed MSP courses, if a participant completed only one course and did not
have a HOUSSE plan it was not possible to determine whether that participant became highly qualified
prior to or as a result of MMSP course participation.

Table 20 presents the highly qualified status of participants across all years of the program. This table
provides an unduplicated count of participants. Because the number of courses a participant took was
relevant to the process used to determine how many participants attained highly qualified status while



      UMass Donahue Institute
      Research and Evaluation Group                                                                           37
Annual Report of the MMSP State-level Evaluation                        MMSP Participant Background Data

participating in MMSP, the findings are primarily organized according to number of courses taken
(only one vs. multiple). In Appendix I, a more detailed version of the table is presented in which the
data are further broken down by year of participation and cohort.


         Table 20: Highly Qualified Status of Unique Participants

                                      Took only one course      Took multiple courses      TOTAL
         Status
                                            n = 555                   n = 354              N = 909

         Became Highly Qualified               19                         92                 111

         Highly Qualified in some
                                               17                         19                  36
         content areas but not all
         Highly Qualified but
         unable to determine when             321                        166                 487
         became HQ

         Not Highly Qualified                  83                         23                 106

         Private school or
                                               34                         17                  51
         not teaching

         Unknown                               81                         37                 118


For the first three years of the program, at least 111 participants attained highly qualified status. (This
was 13% of all participants who were public school teachers.) It is likely that more participants gained
highly qualified status between the time they took their first course and completed their most recently
completed course, but the survey approach did not permit capturing the time when all participants
became highly qualified. Of the 111 who attained highly qualified status, 70 did so by completing a
sufficient number of PDPs on their HOUSSE plans, 16 did so by passing the appropriate MTEL, five
did so by obtaining a degree in content areas, three did so by obtaining undergraduate equivalents in
content areas, and three did so by earning a teaching license. In addition, 14 participants did so by
meeting two or more criteria simultaneously. An identification of the criteria by which teachers who
gained highly qualified status demonstrated competency in their subject matter may be found in
Appendix J.

Table 21 shows the minimum number of participants who gained highly qualified status within each
partnership. Again, it is likely that more participants gained highly qualified status than are indicated.




      UMass Donahue Institute
      Research and Evaluation Group                                                                           38
Annual Report of the MMSP State-level Evaluation                                  MMSP Participant Background Data




                    Table 21: Number of Unique Participants Gaining Highly
                    Qualified Status, by Partnership
                                                                            Number of Unique
                                                                           Participants Gaining
                    Partnership
                                                                             Highly Qualified
                                                                                  Status*
                    EduTron                                                            13

                    Harvard Graduate School of Education                               14

                    Lesley University                                                  36

                    MCLA – Science                                                      3

                    Salem State College                                                17

                    Springfield Public Schools                                         13

                    Wareham Public Schools                                              4

                    Worcester Polytechnic Institute                                     9

                    MCLA – Mathematics                                                  1

                    University of Massachusetts Amherst                                 4

                    TOTAL                                                          111**
                   * Because it was not possible to determine when highly qualified status was
                   attained for a large number of participants, these figures are probably lower than
                   actuality.
                   ** 3 participants are each counted twice as they took courses in two partnerships



MTEL Information
One method by which teachers may demonstrate subject-matter competency is to pass the
Massachusetts Tests for Educator Licensure (MTEL) in the content areas that they teach. Table 22
identifies the tests taken by public school teachers in each year of the study along with passage rates. Of
the 354 participants taking multiple courses, 13% passed an MTEL test and 11% earned new licenses
by the end of the third year of MMSP.

As shown in Table 22, based on data from the last survey completed by each participant, of the 92
participants who had taken the Mathematics MTEL, 70 (76%) reported passing the test, and two (2%)
had not yet received their scores at the time of survey completion. Of the 190 respondents who had
taken the Middle School Mathematics MTEL, 161 (85%) passed and nine (5%) had not yet received
their scores. Of the 25 participants who completed the Middle School Mathematics/Science MTEL,
fifteen (60%) passed and four (16%) had not yet received their scores. Of the 35 participants
completing the General Science MTEL, 31 (89%) passed. Three participants both took and passed the
Chemistry MTEL, and two both took and passed the Earth Science MTEL. Eight respondents attempted
the Biology MTEL, and five (63%) passed. One individual completed the Technology/Engineering
MTEL, but did not indicate whether or not it was passed. Of the 80 participants who reported taking
General Curriculum (formerly elementary) MTEL, 73 (91%) reported passing.


      UMass Donahue Institute
      Research and Evaluation Group                                                                                  39
Annual Report of the MMSP State-level Evaluation                      MMSP Participant Background Data


Table 22: MTEL Tests Taken by MMSP Participants – Total to Date
Based on each participant’s last survey
                                                                                    Scores
                                  Taking Test   Passing Test    Failing Test                    No Response*
                                                                                  Unknown
                                      n          n      %        n       %        n       %       n         %
General Curriculum
                                       80        73     91%       5       6%       0      0%       2        3%
    (formerly Elementary)
Elementary Mathematics                 15        12     80%       0       0%       3     20%       0        0%
Mathematics                            92        70     76%      17      18%       2      2%       3        3%
Middle School Mathematics             190       161     85%      15       8%       9      5%       5        3%
Middle School
                                       25        15     60%       4      16%       4     16%       2        8%
Mathematics/Science
General Science                        35        31     89%       1       3%       1      3%       2        6%
Biology                                 8          5    63%       1      13%       0      0%       2       25%
Chemistry                               3          3   100%       0       0%       0      0%       0        0%
Physics                                 0          0     0%       0       0%       0      0%       0        0%
Earth Science                           2          2   100%       0       0%       0      0%       0        0%
Technology/Engineering                  1          0     0%       0       0%       0      0%       1       100%

TOTAL in STE Areas                     74        56     76%       6       8%       5      7%       7        9%



HOUSSE Professional Development Plan Status
Teachers may demonstrate subject-matter competency through a HOUSSE plan (see Appendix H). As
shown in Table 23, at the end of the third year of the program, many public school participants (27%)
were unsure whether or not they had a HOUSSE plan, and an additional 4% did not answer that survey
question. Only 44% of the 354 participants who took more than one course were consistent across
surveys in their responses to this item, indicating that many were confused about the issue. (All survey
respondents (not just public school teachers) had been asked to respond to the item on HOUSSE plan
status.) In all, only 38% of respondents working in public schools indicated that they had HOUSSE
plans.

According to the guidelines established by the MADOE, in the 2003-2004 academic year, teachers
needed 48 or more PDP hours in the content areas they teach to be considered highly qualified. For the
2005-2006 academic year, teachers were required to complete at least 96 content hours of their
HOUSSE plan to attain highly qualified status. Information regarding PDPs was requested from only
those participants who reported that they had a HOUSSE plan. At the end of the third year of the
program, of the 332 public school teachers indicating they had a HOUSSE plan, 76% reported having
earned 48 or more PDP hours, 10% reported having fewer than 48 PDP hours, and 14% did not respond
to the question.




      UMass Donahue Institute
      Research and Evaluation Group                                                                               40
        Annual Report of the MMSP State-level Evaluation                                            MMSP Participant Background Data




    Table 23: HOUSSE Plan Status of Unique Public School Teachers

                                                                           Year 1            Year 2             Year 3             Total to Date
                                                                          N        %       N          %        N       %            N        %
    Do you have a High Objective Uniform State                Yes          75      23%     177         40%     196     43%           332     38%
    Standard of Evaluation (HOUSSE) plan?                      No         142      43%     133         30%     137     30%           285     32%
                                                      Not Sure             98      30%     127         28%     106     23%           237     27%
                                                  No Response              17       5%         9        2%      16       4%           29      3%
    If you do have a HOUSSE           Fewer than 48 PDP hours                 9    12%       11         6%      17       9%           32     10%
    Plan, how many PDP hours do             48 to 96 PDP hours                6     8%       19        11%      13       7%           22      7%
    you have in your content area(s)?     97 or more PDP hours             46      61%     115         65%     136     69%           231     70%
                                                  No Response              14      19%       32        18%      30     15%            47     14%



        Mathematics, Science and Technology Licensure in Content Area Taught
        By the end of Year 3, 147 regular education teachers and 22 special education teachers reported
        teaching in science or technology areas. For regular education teachers, the licensing reported by
        approximately 35% appeared to be appropriate for the level taught. For special education teachers, the
        licensing reported by approximately 3% appeared to be appropriate for the level taught. Tables 24a and
        24b show how many teachers taught in each science and technology area for Year 1, Year 2, Year 3,and
        over the course of all years, and the tables show how many teachers reported being licensed in the area
        in which they taught. Table 24a provides information for regular education teachers and Table 24b
        provides information for special education teachers. (The numbers presented in Tables 24a and 24b
        exceed the number of teachers who reported teaching in these areas because some teachers taught in
        more than one area.)



Table 24a: MMSP Science and Technology Teaching Areas—Regular Education

                                Year 1                              Year 2                            Year 3                         Total
                     Teach in     Licensed in        Teach               Licensed in       Teach        Licensed in      Teach       Licensed in Area
Content Area
                       Area       Area Taught       in Area              Area Taught       in Area      Area Taught    in Area            Taught
                        n        n        %*           n             n            %*           n        n       %*         n            n      %*
General Science        49        20       40.8%       58            28             48.3%       46       20     43.5%      103        32        31.1%
Biology                17        12       70.6%       17            12             70.6%       13       11     84.6%       29        21        72.4%
Earth Science          17         0        0.0%       11             0              0.0%       10         1    10.0%       22           1        4.5%
Physics                11         2       18.2%       11             0              0.0%        9         2    22.2%       15           2      13.3%
Chemistry              11         3       27.3%        9             1             11.1%       10         5    50.0%       17           8      47.1%
Technology              9         0        0.0%        1             1            100.0%        4         3    75.0%           8        3      37.5%
        *Of the number of regular education participants teaching in this area for this year




                  UMass Donahue Institute
                  Research and Evaluation Group                                                                                                         41
        Annual Report of the MMSP State-level Evaluation                                                            MMSP Participant Background Data



  Table 24b: MMSP Science and Technology Teaching Areas—Special Education

                                 Year 1                                Year 2                                   Year 3                                            Total
  Content             Teach       Licensed in           Teach           Licensed in              Teach              Licensed in                   Teach                Licensed in
  Area                in Area     Area Taught           in Area         Area Taught             in Area             Area Taught                   in Area              Area Taught
                        n          n           %*         n              n            %*          n                 n                 %*            n                  n              %*
  Gen. Science            10           1       10.0%          8              1        12.5%            7                 0            0.0%           16                    0          0.0%
  Biology                   4          0        0.0%          2              0        0.0%             3                 0            0.0%              7                  0          0.0%
  Earth Science             4          0        0.0%          1              0        0.0%             4                 0            0.0%              5                  0          0.0%
  Physics                   0          0        0.0%          1              0        0.0%             1                 0            0.0%              1                  0          0.0%
  Chemistry                 1          0        0.0%          1              0        0.0%             1                 0            0.0%              2                  1         50.0%
  Technology                0          0        0.0%          0              0        0.0%             0                 0            0.0%              0                  0          0.0%
   * Of the number of special education participants teaching in this area for this year
   ** The “Total” number may be smaller than the sum of Years 1 through 3 because the Total is calculated from each participant’s
   last completed survey, and individuals may have changed teaching areas or not reported licensure in a given area on that survey.



        By the end of Year 3, 426 regular education teachers and 69 special education teachers reported
        teaching mathematics. For regular education teachers, the licensing reported by approximately 67%
        appeared to be appropriate for the level taught. For special education teachers, the licensing reported by
        approximately 18% appeared to be appropriate for the level taught. Tables 25a and 25b provide a
        breakdown of each math level for Year 1, Year 2, Year 3, and for the total over the course of all years,
        with Table 25a including information for regular education teachers and Table 25b including
        information for special education teachers.



Table 25a: MMSP Mathematics Teacher Levels —Regular Education

                                 Year 1                                  Year 2                                         Year 3                                                 Total
                      Teach      Licensed in Area         Teach              Licensed in              Teach             Licensed in Area                    Teach              Licensed in Area
Content Area
                      in Area        Taught              in Area             Area Taught              in Area               Taught                          in Area                 Taught
                        n           n           %*            n              n             %*              n                 n               %*               n                  n           %*
Middle School            122            57      46.7%         187            103         55.1%             178               131           73.6%               354               223         63.0%
High School                 12             9    75.0%             17             13      76.5%                 39                36        92.3%                  59                 52      88.1%
MS & HS grades              1              1   100.0%              3              2      66.7%                  1                 0          0.0%                  4                  3      75.0%
TOTAL Math               135            67     49.6%          207            118        57.0%              218               167           76.6%               417               278         66.7%
   *Of the number of regular education participants teaching in this area for this year




                  UMass Donahue Institute
                  Research and Evaluation Group                                                                                                                                                      42
        Annual Report of the MMSP State-level Evaluation                                           MMSP Participant Background Data



Table 25b: MMSP Mathematics Teacher Levels —Special Education

                               Year 1                            Year 2                            Year 3                         Total
                    Teach      Licensed in Area      Teach         Licensed in        Teach         Licensed in Area   Teach      Licensed in Area
Content Area
                    in Area        Taught            in Area       Area Taught       in Area            Taught         in Area         Taught
                       n         n         %*           n          n        %*            n           n        %*         n         n         %*
Middle School           20           2     10.0%            23         4   17.4%              32          5    15.6%       53           7    13.2%
High School                3         1     33.3%             4         3   75.0%               3          1    33.3%          7         3    42.9%
MS & HS grades             0         0     0.0%              3         1   33.3%               1          0     0.0%          2         1    50.0%
TOTAL Math              23           3    13.0%             30         8   26.7%              36          6   16.7%        62           11   17.7%
   *Of the number of special education participants teaching in this area for this year




                UMass Donahue Institute
                Research and Evaluation Group                                                                                                        43
Annual Report of the MMSP State-level Evaluation                         MMSP Participant Background Data

Goal III.
            Increase the number of highly qualified teachers in mathematics, science, and/or
            technology/engineering by integrating the courses of study into schools of education and/or
            arts and sciences at institutes of higher education.

For systemic change to occur at the higher education institutions, Departments of Arts and Sciences and
Education Departments are encouraged to work together through MMSP to support stronger content
courses in mathematics and science for teacher preparation, undergraduate and graduate degree
requirements, and for in-service teachers pursuing graduate-level content courses for recertification.
Integration of Title IIB courses into graduate programs at Institutes of Higher Education will ensure
sustainability over time. The intent behind encouraging the partnerships is that the faculty from the Arts
and Sciences Departments will bring strong content expertise to the partnership table. This integration
will create greater opportunities for participants to complete coursework leading to a content-area
degree and/or to licensure along with the highly qualified designation.

While this report focuses on Year 3 of the MMSP, institutional shift of the type the MMSP intends to
promote requires time to occur and build upon itself over time. For Year 3, because partnerships were
asked to report on integration activities only for Year 3 (and not for the entire duration of their projects)
in order to convey as thorough of a sense as possible of the impact the MMSP has had on
institutionalization, information related to it that has been gathered over the course of all three years
will be presented here, chronologically, for each individual year.

At the end of Year 1, partnerships reported very general information about integration, or they reported
specific percentages of integration into higher education institutions that were unclear. All partnerships
reported progress toward meeting this goal, though. At the end of Year 1, the following five
partnerships reported that they had integrated at least one course into their higher education institute:
Salem, Lesley, Wareham, Springfield, and WPI.

For Year 2, partnerships were asked to report more detailed information regarding how integration of
the Title IIB courses was occurring, and they were provided with more guidance about what to report.
Specifically, they were asked to complete the following five items in the context of an Annual Report
Addendum:

    1. Which course(s), if any, have been embedded into existing undergraduate or graduate programs
       at your MMSP higher education partner? Please specify the department (education, math,
       science) into which the course(s) have been embedded.
    2. Which course(s), if any, have been integrated into the Continuing education program at your
       MMSP higher education partner? Please describe the types of course credit and/or PDPs
       available to those taking the course through the continuing education program.
    3. Has any part of your MMSP Course curricula (i.e., case studies, assessments, materials) been
       integrated into other existing programs or courses at your MMSP higher education partner? If
       yes, please describe.
    4. Has your MMSP higher education partner created a new degree program based on the courses
       developed for your MMSP program? If yes, please describe.
    5. Describe any other steps you have made to integrate your MMSP course of study at your
       MMSP higher education partner.

Integration or plans for integration in the future were evident from the nine partnerships that were
actively providing courses during Year 2. As would be expected in a program involving partnerships
with diverse structures and styles, the extent and type of integration varied across partnerships. To


      UMass Donahue Institute
      Research and Evaluation Group                                                                             44
Annual Report of the MMSP State-level Evaluation                        MMSP Participant Background Data

convey a sense of how integration occurred, following are summaries for each partnership that was
actively providing courses.

         All five MMSP courses that EduTron created were offered through the Center for Professional
          Studies at Fitchburg State College. All courses were offered to in-service teachers for graduate
          credit and PDPs.

         Harvard used some of the case studies that were created through MMSP with the non-MMSP
          math and science teacher education students at the Harvard Graduate School of Education.

         Prior to MMSP funding, Lesley University did not have a Math or Science Department, but as a
          result of the availability of Title IIB funds, they brought Dr. Ken Gross from the Vermont
          Mathematics Initiative to Lesley to create a brand new Lesley University Center for
          Mathematics Achievement, and, therefore, built Lesley's capacity by expanding their math
          faculty. This also caused institutional change by creating several new degree programs
          including the following: two new master's degrees with a specialization in elementary or middle
          mathematics education; a CAGS program for mathematics education to accommodate those
          teachers already possessing master’s degrees; and three new faculty positions.

         The physics course for middle school teachers developed by Massachusetts College of Liberal
          Arts (MCLA) Science was offered as part of the undergraduate teacher education curriculum.

         During Year 1 of the program, Salem State College developed a new master’s level teaching
          program in middle school mathematics. All courses developed by Salem State College through
          MMSP can be applied towards earning a degree through that program.

         The four science courses offered by the Springfield partnership through the MMSP program
          have been offered for graduate credit through University of Massachusetts Amherst (UMass
          Amherst) Continuing Education. In addition, the chemistry course offered through the
          partnership also was offered for graduate level credit through the Chemistry Department and
          both the chemistry and earth science courses were offered for graduate level credit in the
          master’s level Science Education Program at the School of Education at UMass Amherst.
          Materials, activities, and assessments have been used in UMass Amherst Science Education
          Online courses and in a course at Hampshire College.

         The two math courses developed through the UMass Amherst partnership can be used to fulfill
          requirements for the UMass Amherst Masters of Education degree and for the CAGS. In
          addition, they help to fulfill the subject matter requirements for initial licensure in middle
          school mathematics for the State and National Council for Accreditation of Teacher Education
          approved UMass Secondary Teacher Education Program.

         All three math courses developed through the Wareham partnership have been embedded into
          existing programs at UMass Dartmouth. They are part of the mathematics graduate and
          undergraduate programs and also part of the master’s level teaching program. Courses are
          available to teachers working toward certification and to certified teachers seeking advanced
          degrees and recertification.

         The four courses created by WPI were embedded into a master’s level graduate program in
          mathematics education offered by WPI’s Mathematics Department. MMSP funding also aided
          WPI in expanding their master’s level graduate program in mathematics education to include
          middle school level courses when it previously had only included high school level courses.


        UMass Donahue Institute
        Research and Evaluation Group                                                                        45
Annual Report of the MMSP State-level Evaluation                        MMSP Participant Background Data

In addition, to the integration activities that are already occurring as a result of MMSP efforts, four
partnerships articulated specific plans for additional future integration activities.

For Year 3, partnerships were asked to describe the activities of their partnerships during the Year 3
funding period that spoke to the “institutionalization” of their courses, the extent to which their courses
have been integrated into activities of their higher education partners. The extent and type of integration
varied across partnerships. Following are summaries for partnerships that indicated that additional
integration was occurring in Year 3:

         In Year 3, all MMSP courses that EduTron created were offered through the Center for
          Professional Studies at Fitchburg State College. All courses were offered to in-service teachers
          for graduate credit. In addition, the partnership united Mathematics faculty and Education
          faculty at Fitchburg State College in a fundamental way: All parties are now working together
          to improve teacher preparation in mathematics through more stringent requirements in math
          courses and by improving mathematics offerings. The education department is in the process of
          hosting a math education summit to articulate the problems and present potential solutions that
          are arrived at through the joint efforts of math and education faculty.

         In addition to continuing with the programs that were created in prior MMSP years, Lesley
          University has begun developing an online mathematics program that will offer a master’s
          degree in elementary or middle school mathematics education. Through needs that have arisen
          through the classroom implementation component of their participation in MMSP, Lesley also
          has developed a relationship with Cisco Corporation who will be providing the online platform
          for the online mathematics program.

         Massachusetts College of Liberal Arts (MCLA) Science now offers two new variable credit
          science for educators courses for in-service teachers.

         Massachusetts College of Liberal Arts (MCLA) Math now offers one new variable credit math
          for educators courses for in-service teachers.

         Salem State College continues to offer courses developed through MMSP as part of a master’s
          level teaching program in middle school mathematics. All courses developed by Salem State
          College through MMSP can be applied towards earning a degree through that program.

         The three science courses offered by the Springfield partnership through the MMSP program in
          Year 3 were offered for graduate credit through UMass Amherst Continuing Education. A goal
          of the partnership is to have these courses institutionalized in the UMass Amherst School of
          Continuing Education through the School of Education.

         All courses developed through the UMass Amherst partnership can be taken for graduate level
          credits that can be transferred to a variety of graduate level programs at UMass Amherst. In
          addition, the two courses are being reviewed by the School of Education Academic matters
          Committee for permanent course approval.

         The one new course created by WPI in Year 3 was embedded into the master’s level graduate
          program in mathematics education offered by WPI’s Mathematics Department.




        UMass Donahue Institute
        Research and Evaluation Group                                                                         46
Annual Report of the MMSP State-level Evaluation                          MMSP Participant Background Data

Goal IV.
           Increase the number of MSTE teachers currently employed in the partner school districts who
           participate in content-based professional development activities and substantially increase their content
           knowledge in order to be able to teach effectively the state learning standards.

Content Knowledge Gains
As a grant condition, MMSP partnerships were required to create a pre-course test and post-course test to assess
participants’ knowledge of the content for each MMSP course. In most cases, the faculty who developed the
course also developed the assessment. For each course, the same instrument served as both the pre-course test and
the post-course test. Due to time and resource constraints, partnerships were not required to test their assessments
for validity or reliability. Also, partnerships were not encouraged to locate a previously existing standardized
instrument that had demonstrated validity and reliability because a priority was placed on developing assessments
that would reflect the precise content that would be taught in each of their courses.

Eighty-four of the 85 courses offered across all partnerships to date showed score gains in average percentage of
items correct between pre- and post-course test administrations. (Data were analyzed only for participants who
completed both pre- and post-course assessments.) To determine if the scores showed statistically significantly
changes between pre- and post-course test administrations, a paired samples t-test was used for each course for
which ten or more participants completed both pre- and post-course assessments, and a Wilcoxon matched pairs
test was used for courses for which fewer than ten participants completed both pre- and post-course assessments.
Statistically significant improvements in scores on content knowledge assessments occurred in 93% of all courses
offered through MMSP courses. Table 26 provides information on pre- and post-course average scores, pre-to-
post changes in average scores, and statistical significance of differences between pre-course and post-course
scores.

Three courses offered through one partnership had both treatment (i.e., MMSP) and comparison groups complete
the content knowledge assessments. In each course, the comparison group was tested only once. For each course,
the treatment group post-course scores were statistically significantly higher than those of the comparison group,
although in one case, the treatment group pre-course scores also were statistically significantly higher than those
of the comparison group.



        Table 26: Mean Percentage Scores for Pre-course & Post-course Tests, Including Gains
        in Mean Scores

         Year                                                   Mean          Mean       Change on
                         Institute Name – Course         N                                            p <.05
        Offered                                                Pre-test      Post-test     Mean

                                                               Arit: 72      Arit: 85     Arit: 13   Arit: Yes
            1     EduTron – Building a Solid Bridge      37
                                                               Alg: 26       Alg: 56      Alg: 30    Alg: Yes

                                                               Conc: 12      Conc: 60    Conc: 48    Conc: Yes
            1     EduTron – Linear Equations             39
                                                               Com: 46       Com: 66     Com: 20     Com: Yes

                  EduTron – Building a Rock Solid Math
            2                                            26       46            59           13        Yes
                  Foundation

                  EduTron – An Odyssey to Algebra,
            2                                            31       61            78           17        Yes
                  Geometry and Back

                  EduTron – A Further Odyssey to
            2                                            31       54            80           25        Yes
                  Algebra, Geometry and Back




      UMass Donahue Institute
      Research and Evaluation Group                                                                                47
Annual Report of the MMSP State-level Evaluation                            MMSP Participant Background Data


        Table 26: Mean Percentage Scores for Pre-course & Post-course Tests, Including Gains
        in Mean Scores
         Year                                                     Mean          Mean       Change on
                         Institute Name – Course            N                                          p <.05
        Offered                                                  Pre-test      Post-test     Mean

                  EduTron – Perspectives on Elementary
           3                                                20      51            73          22        Yes
                  Mathematics

                  EduTron – Perspectives on Middle
           3                                                38      60            84          23        Yes
                  School Mathematics

           1      Harvard – Mathematics Case Study          23      54            72          18        Yes

           2      Harvard – Making the Case 1b              27      52            63          12        Yes

           2      Harvard – Making the Case 1c              24      65            75          10        Yes

           2      Harvard – Making the Case II              15      74            81           7        Yes

           3      Harvard – Making the Case I               17      50            65          15        Yes

           3      Harvard – Making the Case I (SPED)        27      32            47          15        Yes

           3      Harvard – Making the Case I               32      57            64           7        No

                  Lesley – Math as a Second Language,
           1      Cohort 1                                  22      82            87           5        Yes

                  Lesley – Math as a Second Language,
           1      Cohort 2                                  26      69            76           7        Yes

                  Lesley – Math as a Second Language,
           1      Cohort 3                                  37      73            83          10        Yes

                  Lesley – Functions and Algebra,
           1      Cohort 1                                  16      64            81          17        Yes

                  Lesley – Functions and Algebra –
           2      cohort 2                                  18      55            79          24        Yes

                  Lesley – Functions and Algebra –
           2      cohort 3                                  39      53            75          22        Yes

                  Lesley – Trigonometry, Algebra, and
           2      Analytic Geometry – cohort 1              31      17            58          41        Yes

                  Lesley – Trigonometry, Algebra, and
           2      Analytic Geometry – cohort 2              44      12            59          48        Yes


           2      Lesley – Number Theory – cohort 1         36      29            79          50        Yes

           2      Lesley – Number Theory – cohort 2         29      39            75          36        Yes

                  Lesley – Statistics and Data Analysis –
           2      cohort 1                                  32      39            84          45        Yes

                  Lesley – Statistics and Data Analysis –
           2                                                15      29            79          51        Yes
                  cohort 2




      UMass Donahue Institute
      Research and Evaluation Group                                                                             48
Annual Report of the MMSP State-level Evaluation                            MMSP Participant Background Data


        Table 26: Mean Percentage Scores for Pre-course & Post-course Tests, Including Gains
        in Mean Scores
         Year                                                     Mean          Mean       Change on
                         Institute Name – Course            N                                          p <.05
        Offered                                                  Pre-test      Post-test     Mean

                  Lesley – Measurement and Probability
           3                                                21      36            74          38        Yes
                  – cohort 1

                  Lesley – Measurement and Probability
           3                                                44      42            71          29        Yes
                  – cohort 2

                  Lesley – Statistics and Data Analysis –
           3                                                21      28            59          31        Yes
                  cohort 3

           3      Lesley – Calculus – cohort 1              25      27            61          34        Yes


           3      Lesley – Calculus – cohort 2              27      35            75          40        Yes


           3      Lesley – Calculus II – cohort 1            9      35            65          29        Yes

                  MCLA Science - Conceptual Physical
           2                                                14      38            55          17        Yes
                  Science

           3      MCLA Science – Conceptual Biology         10      46            55           9        Yes

           1      Salem State – Statistics                  23      60            79          19        Yes

                  Salem State – Geometry (spring
           1      session)                                  16      20            82          62        Yes


           1      Salem State – Number Theory               13      65            71           6        No

                  Salem State – History of Math (June
           1                                                 7      21            81          61        Yes
                  session)

                  Salem State – Geometry (summer
           1                                                 9      39            84          45        Yes
                  session)

                  Salem State – History of Math (July
           1      session)                                  17      59            82          23        Yes

                  Salem State – Patterns, Relations, and
           2                                                15      61            80          19        Yes
                  Algebra for Middle School Teachers

                  Salem State – Data, Probability, and
           2                                                11      55            89          34        Yes
                  Statistics for Middle School Teachers

                  Salem State – Number Systems for
           2                                                29      39            76          36        Yes
                  Middle School Teachers

                  Salem State – Patterns, Relations, and
           2                                                16      78            86           8        Yes
                  Algebra for Middle School Teachers

                  Salem State – PreCalculus for Middle
           2                                                20      49            79          29        Yes
                  School Teachers




      UMass Donahue Institute
      Research and Evaluation Group                                                                             49
Annual Report of the MMSP State-level Evaluation                             MMSP Participant Background Data


        Table 26: Mean Percentage Scores for Pre-course & Post-course Tests, Including Gains
        in Mean Scores
         Year                                                      Mean          Mean       Change on
                         Institute Name – Course             N                                          p <.05
        Offered                                                   Pre-test      Post-test     Mean

                  Salem State – History of Math for
           2                                                 11      24            89          66        Yes
                  Middle School Teachers

                  Salem State – Number Systems for
           2                                                  7      73            94          21        Yes
                  Middle School Teachers

                  Salem State – Discrete Mathematics for
           2                                                 12      13            83          71        Yes
                  Middle School Teachers

                  Salem State – Geometry and
           2      Measurement for Middle School               6      36            81          45        Yes
                  Teachers

                  Salem State – Calculus for Middle
           2                                                 13      22            66          44        Yes
                  School Teachers

                  Salem State – Patterns, Relations, and
           3                                                 17      72            86          14        Yes
                  Algebra for Middle School Teachers

                  Salem State – Linear Systems for
           3                                                 20       8            87          79        Yes
                  Middle School Teachers

                  Salem State – Precalculus for Middle
           3                                                  8      60            85          25        Yes
                  School Teachers

                  Salem State – Precalculus for Middle
           3                                                 21      34            89          55        Yes
                  School Teachers

                  Salem State – Calculus for Middle
           3                                                 12      37            69          32        Yes
                  School Teachers

                  Salem State – Discrete Mathematics for
           3                                                  9      13            82          69        Yes
                  Middle School Teachers

                  Salem State – Geometry and
           3      Measurement for Middle School              17      22            68          45        Yes
                  Teachers

                  Salem State – Number Systems for
           3                                                 17      20            76          56        Yes
                  Middle School Teachers

                  Salem State – History of Math for
           3                                                 14      24            90          65        Yes
                  Middle School Teachers

                  Salem State – Probability and Statistics
           3                                                 19      15            88          72        Yes
                  for Middle School Teachers

           1      Springfield Public Schools – Physics       20      23            38          15        Yes

           1      Springfield Public Schools – Chemistry     24      53            87          34        Yes

                  Springfield Public Schools – Life
           2                                                 25      48            71          23        Yes
                  Science



      UMass Donahue Institute
      Research and Evaluation Group                                                                              50
Annual Report of the MMSP State-level Evaluation                            MMSP Participant Background Data


        Table 26: Mean Percentage Scores for Pre-course & Post-course Tests, Including Gains
        in Mean Scores
         Year                                                     Mean          Mean       Change on
                         Institute Name – Course            N                                          p <.05
        Offered                                                  Pre-test      Post-test     Mean

                  Springfield Public Schools – Earth
           2                                                31      53            64          11        Yes
                  Science

                  Springfield Public Schools –
           3                                                26      40            79          39        Yes
                  Mathematics for Science Teachers

                  Springfield Public Schools –
           3                                                23      38            74          36        Yes
                  Technology/Engineering

                  Springfield Public Schools – Integrated
           3                                                34      33            71          38        Yes
                  Science

                  Wareham Public Schools – Improving
           1                                                37      67            67           0        No
                  Math Teaching

                  Wareham Public Schools – Topics in
           2      Mathematics for Teachers: Improving        8      65            76          11        No
                  Math Teaching Part II

                  Wareham Public Schools – Developing
           2                                                 7      74            80           6        No
                  Teacher Portfolios

                  Worcester Polytechnic Institute –
           1                                                12      52            78          26        Yes
                  Geometry

                  Worcester Polytechnic Institute –
           1                                                 6      26            47          20        Yes
                  Discrete Mathematics

                  Worcester Polytechnic Institute –
           2                                                57      55            77          23        Yes
                  Algebra for Middle School Teachers

                  Worcester Polytechnic Institute –
           2                                                22      41            65          24        Yes
                  Probability and Statistics

                  Worcester Polytechnic Institute –
           3      Middle School Mathematics and             62      50            58           8        Yes
                  MCAS Studies

                  Worcester Polytechnic Institute –
           3                                                26      36            58          22        Yes
                  Geometry

                  MCLA Math – Problem Solving and
           3                                                 6      59            75          16        Yes
                  Numeration

                  MCLA Math – Patterns, Relations, and
           3                                                 8      17            57          40        Yes
                  Algebra

                  University of Massachusetts Amherst -
           2                                                14      28            94          66        Yes
                  Algebra – cohort 1

                  University of Massachusetts Amherst -
           2                                                20      52            77          26        Yes
                  Geometry – cohort 1




      UMass Donahue Institute
      Research and Evaluation Group                                                                             51
Annual Report of the MMSP State-level Evaluation                          MMSP Participant Background Data


        Table 26: Mean Percentage Scores for Pre-course & Post-course Tests, Including Gains
        in Mean Scores
         Year                                                   Mean          Mean       Change on
                         Institute Name – Course          N                                          p <.05
        Offered                                                Pre-test      Post-test     Mean
                  University of Massachusetts Amherst -
           3                                              20      41            72          32        Yes
                  Algebra – cohort 2

                  University of Massachusetts Amherst -
           3                                              15      60            84          24        Yes
                  Geometry – cohort 2

                  University of Massachusetts Amherst –
           3                                               4      41            94          54        No
                  Algebra B (mini course)

                  University of Massachusetts Amherst -
           3                                              13      51            93          42        Yes
                  Trigonometry – cohort 1

                  University of Massachusetts Amherst –
           3                                              17      30            90          61        Yes
                  Discrete Mathematics – cohort 1




      UMass Donahue Institute
      Research and Evaluation Group                                                                           52
Annual Report of the MMSP State-level Evaluation                     MMSP Participant Background Data



Goal V.
          Improve student academic achievement as measured by Massachusetts Comprehensive Assessment
          System (MCAS) mathematics and science and technology/engineering assessments and other
          assessments.

When possible, partnerships collected student achievement data for their local evaluations. There are two major
challenges to collecting MCAS data for students of participating teachers. The first is that there are legal and
logistical barriers to collecting student-level data from school districts that make it very difficult, if not
impossible, to collect data. The other is that, especially in science, MCAS is not administered each year, so
partnerships are not able to use the MCAS data consistently across years.

Number of Students Taught
By the end of Year 3 of the MMSP, those who participated in MMSP courses had been teaching over 89,000
students. At the end of Year 3, of the 873 participants who responded to a survey question regarding the number
of students taught, 26% reported teaching between 1 and 40 students, 68% reported teaching between 41 and 150
students, and 5% taught over 151 students annually.

Participants also reported the number of Title I, academically advanced, special education, and English language
learners they taught annually. During survey administration, some participants reported to their local evaluators
that they had difficulty estimating the number of these students they taught annually. Therefore, limited
confidence should be placed in this data. Despite concerns with the accuracy of this data, these figures are
reported in the tables of all Participant Background Survey results in Appendix F.




      UMass Donahue Institute
      Research and Evaluation Group                                                                                 53
Annual Report of the MMSP State-level Evaluation                                                            Discussion




  Discussion and Recommendations

The MMSP partnership activities described in this report occurred between February 2, 2004 and August 31,
2006. The progress partnerships made at delivering programs is due in large part to the efforts of the partnerships
and their local evaluators, and to the work done at the MADOE to support these programs.

Progress has been made towards meeting both the formal and informal goals of the program. The following
sections highlight areas where progress has been made and also areas and issues that would benefit from some
attention and effort. Specific recommendations are provided for follow-up activities.

Data Quality
While in many instances for this project the data are sound, there are some areas for which this was not the case.
The Donahue Institute became aware that problems existed with some of the data when 1) surveys received from
participants taking multiple courses revealed inconsistencies in reporting across surveys for individual
participants, especially for items used in determining highly qualified status, and 2) surveys indicated that many
participants were confused about whether or not they had a HOUSSE plan.

The Participant Background Survey was modified after Year 1 to improve the likelihood of obtaining good
quality data, but similar problems with the data arose again in Year 2. The survey was modified again in Year 3,
but some problems still existed, and totals to date will be affected by problems from Year 1 and Year 2; therefore,
information regarding the highly qualified status of teachers must be viewed with caution.

Reaching Targeted Participants
As a whole, the partnerships exceeded the MADOE goal of course enrollment consisting of at least 50% of
participants from high need districts, and of the ten partnerships providing courses by the end of Year 3, five of
them had met that goal as individual partnerships.

The goal to reach the participants who need professional development most was emphasized in summer 2005 by
the MADOE in meetings with partnerships. The intended outcome for doing this was to increase the level of
participation from high need districts. Two obstacles existed for partnerships when trying to raise the high need
district participation. The first was that some participating districts that had been designated as high need districts
at the start, when partnerships were formed, were no longer designated as such in Year 2. It was clear that the
commitment to the district was a priority to both MADOE and the partnerships, so there was no way to remediate
this issue. The other obstacle that existed involved one partnership, in particular, that had committed to one cohort
going through multiple courses over the life of the project. For this partnership, it was not possible to change their
participants mid-program.

         MADOE should continue to remind partnerships that enrollment must include a high percentage of
          participants from high needs districts at the start of the program.

         MADOE should encourage partnerships to recruit additional participants from their high need partners
          and/or identify additional high need partners, even if they meet the minimum participant rate of 50%.

         MADOE has already begun revisiting the procedure by which districts are identified as high need
          districts. MADOE should continue efforts in this direction and should consider developing criteria to
          identify individual schools as high need.



        UMass Donahue Institute
        Research and Evaluation Group                                                                                54
Annual Report of the MMSP State-level Evaluation                                                             Discussion


Local Evaluation Designs
The Minimum Evaluation Requirements document (see Appendix B) has been the guiding document for
providing partnerships with information regarding what they need to do for local and state-level evaluation. This
has been the resource most used when providing technical assistance to partnerships.

It was determined early in the initial meetings with partnerships that implementing a rigorous evaluation design –
using randomized-controlled trials or strong comparison groups – would be difficult, if not impossible. The
limitations that arose were limited resources, the lack of sufficient participant numbers to form reasonable
treatment and control groups, and insufficient time to recruit participants into a control group before the start of
the program. At the end of Year 2, only one partnership had implemented a quasi-experimental evaluation design.
The remaining partnerships implemented basic evaluation models utilizing pre-post comparisons. To address the
difficulties that partnerships have had in creating programs based on a rigorous evaluation design, the RFP for
MMSP programs for 2006-2007 included information on how partnerships might obtain additional support
through MADOE and the Donahue Institute to utilize a rigorous approach.

         MADOE and the Donahue Institute should continue to work closely with partnerships to provide
          technical support and resources to conduct rigorous evaluations in the continuation process.

Collecting Student Data
Perhaps the most pervasively problematic issue regarding evaluation of MMSP projects has been difficulty
accessing student data. This has been both a logistical and legal challenge for evaluators. Logistically, many
school districts do not routinely code their student data by teacher; thus, it is difficult to obtain data aggregated at
the participant level. Furthermore, many partnerships anticipate that they will have trouble enlisting the necessary
cooperation from participating districts, particularly those with few teachers enrolled in MMSP courses. To date,
it appears as if only two partnerships, EduTron and Lesley, have made a concerted effort to obtain student
outcome data at the participant (as opposed to school) level.

The expectation to collect student data was both unclear to and unattainable for most partnerships. This led to
some difficult conversations between the Donahue Institute staff and partnership staff when reviewing these
expectations. Partnerships also encountered problems for meeting the federal reporting requirements regarding
student data.

         MADOE should give partnerships guidance on forging agreements with their partner districts to have
          access to student outcome data for participating teachers.

Integrating Courses into Higher Education Institutions
In the first year of MMSP, efforts to determine the extent to which partnerships were able to integrate courses into
their partner higher education institutions were hampered by difficulties in obtaining relevant data. In the second
and third years of the project, efforts made to define integration and provide partnerships with structured guidance
on reporting progress at integration yielded meaningful data. The data provided evidence that integration is
occurring across most partnerships. While the extent and types of integration that have occurred varied across
partnerships, integration has happened primarily in the following ways that encourage sustainability beyond the
duration of the program: embedding MMSP courses into pre-existing programs, integrating courses into pre-
existing continuing education programs, and creating new degree programs.

         MADOE should encourage partnerships to continue efforts to integrate MMSP courses into degree
          programs of partner higher education institutions.




        UMass Donahue Institute
        Research and Evaluation Group                                                                                  55
Annual Report of the MMSP State-level Evaluation                                                   Discussion


Steering Committee
The Steering Committee has continued to provide guidance and feedback to the MADOE on issues regarding the
MMSP partnerships, on broader issues of professional development and math, science, technology, and
engineering. The MADOE is fully aware how fortunate it is to have such an engaged and invested Steering
Committee for this initiative.

         MADOE should continue the role of the Steering Committee providing advice and guidance and make all
          efforts to follow-up on committee suggestions and requests.




        UMass Donahue Institute
        Research and Evaluation Group                                                                      56
Annual Report of the MMSP State-level Evaluation                                                         Appendix A




  Appendix A: Timeline for State-level Evaluation and TA Activities

MMSP State-level Evaluation and Technical Assistance Activities Timeline

The following is a summary timeline of state-level evaluation and technical assistance activities carried out
between February, 2004, and end of Year 3 of the MMSP.

February 2004 Held Kick-off Meeting for all partnerships and their evaluators at the Department of Education

Spring 2004      Conducted individual partnership meetings with local evaluators and partnership leaders to:

                 Review the Minimum Expectations document along with the local evaluation and data collection
                 plan

                 And

                 Explore potential modifications to implementation plans to create opportunities for experimental
                 or quasi experimental design


Spring 2004      Developed common measures for state-level data collection

June 2004        Attended federal meeting held for MSP projects across the country

Summer 2004 Disseminated and collected end-of-course documents designed to collect course-level data for the
            statewide evaluation

Fall 2004        Conducted individual partnership meetings with local evaluators and partnership leaders to:

                 Review the Minimum Expectations document along with the local evaluation and data collection
                 plan

                 And

                 Review the Federal Reporting document to ensure the partnerships were collecting the data
                 needed to complete that report


Winter 2005      Conducted partnership meetings with the two new partnerships funded in the second round that
                 constitutes Cohort 2 to:

                 Introduce the Minimum Expectations document along with the local evaluation and data
                 collection plan

                 And




      UMass Donahue Institute
      Research and Evaluation Group                                                                              57
Annual Report of the MMSP State-level Evaluation                                                          Appendix A


                 Introduce the Federal Reporting document to ensure the partnerships were collecting the data
                 needed to complete that report

June 2005        Held Technical Assistance Meeting for all partnerships regarding the requirements of the Federal
                 DOE report

The following activities were on going throughout the life of the project:

                 Disseminated and collected end-of-course documents designed to collect course-level data for the
                 statewide evaluation

                 Managed data collected from partnerships at the end of each course

                 Provided technical assistance to partnerships in support of local partnership evaluation efforts

                 Monitored local evaluation plans to see they include both formative and summative research
                 questions and corresponding activities

                 Monitored data collection and analysis around the basic logic model of professional development

                 Served as liaison to the U.S. Department of Education for evaluation and research issues
                 including participation in national meetings and periodic conference calls

                 Met with MADOE MSP Team as needed to support integration of evaluation efforts with
                 program goals

                 Attended MMSP Steering Committee meetings in role of state level evaluator and technical
                 assistance provider.




      UMass Donahue Institute
      Research and Evaluation Group                                                                                 58
Annual Report of the MMSP State-level Evaluation                                                         Appendix B




    Appendix B: Minimum Expectations for Evaluation

The Massachusetts Department of Education (MADOE) has contracted with the UMass Donahue Institute to
conduct the statewide evaluation of the Massachusetts Mathematics and Science Partnerships (MSSP) program. In
addition, MADOE has required that each of the funded partnerships allocate $5,000 to the Donahue Institute to
cover costs associated with providing evaluation-related technical assistance and common data collection
instruments to their partnerships. Accordingly, MADOE has the following minimum expectations for the
individual partnership evaluations as well as partnership cooperation with the statewide evaluation efforts.

    Strong partnership evaluation plans – In contrast to many other grant programs funded through MADOE, it is
     expected that each partnership will develop its own plan for partnership evaluation. Each of these evaluations
     should include both formative and summative research questions with data collection and analysis plans that
     are adequate to address those questions.

     The summative aspects of the evaluation should be designed to address the following outcomes for program
     participants: 1) changes in teacher knowledge and skills; 2) changes in the number of “highly qualified”
     teachers; 3) changes in classroom instruction; and 4) changes in student achievement. For more detail on
     expected measures see the Annual Report bullet beginning on page 3 of this document.

     Although it is generally encouraged, there is no requirement that partnerships engage an external evaluator for
     this effort. However, partnerships that plan to use their own staff for the evaluation should have a clear
     written plan outlining how the evaluation activities will be conducted and who (preferably an individual, not
     an organization) will be responsible for each.

    Prescribed anonymous teacher code – In some cases identified below (course participant knowledge
     assessment; course participant background surveys; annual participant survey), partnerships will be expected
     to provide the Donahue Institute with data on individual participants. Partnerships will be instructed to
     collect such data using a prescribed individual coding system provided by the Donahue Institute. This coding
     system will allow data to be anonymously linked across various instruments and program years, thus enabling
     the statewide evaluation to provide a more complete picture of MMSP impacts.

    Systematic tracking of program participants – It is expected that each partnership will maintain a database (or
     other electronic tracking system) of all program participants including a record of the courses each participant
     completed and when those courses were completed. The specific structure of this database may be
     determined by each individual partnership, but it should be designed to allow for follow-up data collection
     from participants regardless of when they completed their last course in the program. (It is not expected that
     partnerships will be able to follow-up with participants who are no longer employed in their district.) Data
     from this database is primarily for the partnerships’ own program management and evaluation purposes – any
     data requested by the Donahue Institute or MADOE will be in aggregate and/or anonymous formats.

    End of course summary reports – At the end of each course, partnerships are required to submit a basic course
     summary report using a template to be provided by the Donahue Institute. This report will include course
     enrollment and completion rates, a template for submitting the pre/post results of the participant content
     knowledge assessment (described below), and a coversheet for the course participant background surveys
     (described below), which should be returned as part of this packet.

         o   Course participant content knowledge assessment – Partnerships must identify or develop an
             objective tool to assess course participants’ content knowledge gains for each course offered. At a


       UMass Donahue Institute
       Research and Evaluation Group                                                                               59
Annual Report of the MMSP State-level Evaluation                                                          Appendix B


            minimum these assessments should be administered as a pre/post test with participants completing the
            pre-test before receiving any course instruction and the post-test at the end of the course. Partnerships
            are encouraged to consider adding a third administration to measure the impact of any course-related
            follow-up activities (e.g., at the end of the school year). These assessments must yield quantifiable
            results that can be manipulated mathematically to yield mean scores and measurement of change over
            time. Data from these assessments must be collected in a manner that will allow individual
            participant results to be reported to the Donahue Institute using a prescribed anonymous individual
            coding system (described above) that will allow linkage of pre-test, post-test and follow-up scores.
            This code will also allow linkage of the knowledge assessment results to other required data.

        o   Course participant background surveys – Partnerships are required to administer a standard
            participant background survey at the end of each course offered. Similar to the survey that has been
            used to gather information on Summer Content Institutes’ participants, this survey asks about their
            qualifications and professional experience. Because we are interested in tracking changes (and/or the
            absence of changes) over time, this survey must be administered to all participants each time they
            complete a course, regardless of whether they have previously completed the survey.

            The Donahue Institute will provide each partnership with machine-readable surveys that should be
            administered, collected and returned to the Institute for processing. These surveys will include the
            same anonymous individual coding system required for the content knowledge assessment described
            above. Partnerships may also choose to use the information collected through these surveys for their
            own program management and evaluation efforts. If so, they may either make copies to keep for their
            records or request to receive an electronic file of results from the Donahue Institute. It is important to
            note that any electronic file returned by the Donahue Institute will only identify teachers through the
            anonymous codes. Partnerships that wish to maintain this data in records with teachers’ names
            should retain copies for themselves or keep their own master list associating each teacher’s name with
            his/her code. The Donahue Institute is happy to help partnerships think through their specific data
            needs and to plan for related survey administration logistics.


   Course participant feedback surveys – Although not required, partnerships are encouraged to administer and
    collect formative feedback from course participants. The Donahue Institute is happy to provide examples of
    such instruments from other professional development programs and/or offer suggestions for improving those
    developed by individual partnerships.

   Annual survey of all program participants – Each partnership must conduct an annual survey of all
    participants who have completed at least one course during the program. It is expected that partnerships will
    be able to track and obtain cooperation from most individuals who are no longer active participants in the
    program, provided that they are still employed in their district. The Donahue Institute will provide each
    partnership with a bank of common required items that will be used to track changes in participants’
    qualifications. A standard reporting template for those common items will also be provided. This data must
    be collected using the prescribed individual coding system to allow anonymous linkage with other data
    sources. Beyond the common required items, partnerships are welcome to add their own questions to these
    annual surveys. The Donahue Institute is happy to provide feedback and offer suggestions for improving any
    additional items added to the surveys. In particular, we encourage partnerships to add questions that will
    generate self-report data on any changes in teaching practices.

    Please note, there may be cases where annual participant surveys and end of course surveys are administered
    at around the same time. By design, these two surveys are likely to ask some of the same questions. In those
    cases, the Donahue Institute can work with individual partnerships to minimize any duplication in the
    instruments.


      UMass Donahue Institute
      Research and Evaluation Group                                                                                 60
Annual Report of the MMSP State-level Evaluation                                                          Appendix B


   Annual report – Each partnership must submit an annual report to the Massachusetts Department of
    Education. To facilitate meta-analysis of all partnership reports it will be organized around a basic template
    provided by the Donahue Institute, while providing opportunities for partnerships to share additional
    information about their programs. The basic framework of the annual report template will be structured
    around the measures outlined in Form F of MMSP proposal. As such, partnerships should be designing data
    collection and management systems that will allow them to report annually on the following:

        Process measures

        o   Number of partnership meetings and level of partner participation
        o   Number of courses developed
        o   Number of courses delivered and total number of professional development hours available through
            those courses
        o   Describe the extent to which the course of study has been integrated into higher education programs.
        o   Number of participants completing each course (may be measured by the number of teachers
            completing the end of course survey)
        o   Number of in-service program participants (taking one or more courses) during the given year.
            Please note: For this particular measure each participant should only be counted once, regardless of
            the number of courses in which s/he participated.
        Outcomes measures – All partnerships should be able to report this data for each program year as well as
        pre-program baseline data, where available. For programs with experimental or quasi-experimental
        designs, these measures should generally be reported for both treatment and control groups. Any
        exceptions should be discussed with the Donahue Institute well in advance.

        o   The following data should be available through the required questions of the annual participant
            survey. Please note: For these particular measures the number of participants should be the number
            of unique individuals meeting the criteria in the given year. Each participant should only be counted
            once, regardless of the number of courses in which s/he participated.
                    Number of participants taking the MTEL
                    Number of participants passing the MTEL
                    Number of participants becoming licensed in the appropriate content area
                    Number of participants with HOUSSE plans
                    Number of participants making progress on their HOUSSE plans
                    Number of participants completing their house plans
        o   For each course delivered in the given year, the number of participants demonstrating content
            knowledge gain on the pre/post content knowledge assessment and average gain of those participants
        o   Percentage of the students of participating teachers scoring in each of the four MCAS performance
            categories (warning, needs improvement, proficient, advanced) – at applicable grade-levels. (Please
            note: It is not sufficient to report changes for the district as a whole, unless all teachers are
            participating in the professional development offered by the partnership.)
        o   Quantifying the level of achievement among students of participating teachers using other
            assessments for grade levels where content-relevant MCAS is not available. (Please note, it is not
            sufficient to report changes for the district as a whole, unless all teachers are participating in the
            professional development offered by the partnership.)



      UMass Donahue Institute
      Research and Evaluation Group                                                                                  61
Annual Report of the MMSP State-level Evaluation       Appendix C




  Appendix C: Participant Background Survey – Year 1




      UMass Donahue Institute
      Research and Evaluation Group                            62
Annual Report of the MMSP State-level Evaluation   Appendix C




      UMass Donahue Institute
      Research and Evaluation Group                        63
Annual Report of the MMSP State-level Evaluation   Appendix C




      UMass Donahue Institute
      Research and Evaluation Group                        64
Annual Report of the MMSP State-level Evaluation   Appendix C




      UMass Donahue Institute
      Research and Evaluation Group                        65
Annual Report of the MMSP State-level Evaluation       Appendix D




  Appendix D: Participant Background Survey – Year 2




      UMass Donahue Institute
      Research and Evaluation Group                            66
Annual Report of the MMSP State-level Evaluation   Appendix D




      UMass Donahue Institute
      Research and Evaluation Group                        67
Annual Report of the MMSP State-level Evaluation   Appendix D




      UMass Donahue Institute
      Research and Evaluation Group                        68
Annual Report of the MMSP State-level Evaluation   Appendix D




      UMass Donahue Institute
      Research and Evaluation Group                        69
Annual Report of the MMSP State-level Evaluation   Appendix D




      UMass Donahue Institute
      Research and Evaluation Group                        70
Annual Report of the MMSP State-level Evaluation   Appendix D




      UMass Donahue Institute
      Research and Evaluation Group                        71
Annual Report of the MMSP State-level Evaluation   Appendix D




      UMass Donahue Institute
      Research and Evaluation Group                        72
Annual Report of the MMSP State-level Evaluation       Appendix E




  Appendix E: Participant Background Survey – Year 3




      UMass Donahue Institute
      Research and Evaluation Group                            73
Annual Report of the MMSP State-level Evaluation   Appendix E




      UMass Donahue Institute
      Research and Evaluation Group                        74
Annual Report of the MMSP State-level Evaluation   Appendix E




      UMass Donahue Institute
      Research and Evaluation Group                        75
Annual Report of the MMSP State-level Evaluation   Appendix E




      UMass Donahue Institute
      Research and Evaluation Group                        76
Annual Report of the MMSP State-level Evaluation   Appendix E




      UMass Donahue Institute
      Research and Evaluation Group                        77
Annual Report of the MMSP State-level Evaluation   Appendix E




      UMass Donahue Institute
      Research and Evaluation Group                        78
Annual Report of the MMSP State-level Evaluation   Appendix E




      UMass Donahue Institute
      Research and Evaluation Group                        79
Annual Report of the MMSP State-level Evaluation   Appendix E




      UMass Donahue Institute
      Research and Evaluation Group                        80
Annual Report of the MMSP State-level Evaluation                                                                                 Appendix F




  Appendix F: Results of the Participant Background Survey

                              Item                                   Year 1           Year 2                Year 3         Total to Date
                                                                      n    (%)       n     (%)          n       (%)        n        (%)
  How do you describe yourself?
            American Indian or Alaskan native                        0     (0%)        0    (0%)          2    (<1%)         2      (<1%)
            Asian or Pacific Islander                                5     (2%)        6    (1%)          7     (2%)        15       (2%)
            Black – not Hispanic                                     5     (2%)       10    (2%)         15     (3%)        25       (3%)
            Hispanic                                                 6     (2%)       12    (3%)          8     (2%)        20       (2%)
            White                                                  307    (90%)      402   (88%)        404    (87%)       793      (87%)
            Mixed Race                                               2     (1%)        6    (1%)          6     (1%)        11       (1%)
            Other                                                    6     (2%)        6    (1%)          8     (2%)        13       (1%)
            No Response                                             10     (3%)       14    (3%)         14     (3%)        30       (3%)
   What best describes your current position?
           Teacher (Regular Education)                             270 (79%)         351   (77%)        331    (71%)       674     (74%)
           Special Education Teacher (Sole Instructor)              25 (7%)           31    (7%)         43     (9%)        75      (8%)
           Special Education Inclusion Teacher                      22 (7%)           31    (7%)         38     (8%)        65      (7%)
           Other                                                    13 (4%)           23    (5%)         22     (5%)        43      (5%)
           Department Head or Curriculum Coordinator                 5 (2%)           11    (2%)         10     (2%)        22      (2%)
           Principal/Asst. Principal/Headmaster                      3 (1%)            3    (1%)          5     (1%)         9      (1%)
           Support Specialist (counselor, librarian, etc.)           2 (1%)            3    (1%)          5     (1%)         8      (1%)
           Long-term Substitute                                       N/A              1   (<1%)          3     (1%)         2     (<1%)
           Paraprofessional                                          0 (0%)            1   (<1%)          3     (1%)         4     (<1%)
           Superintendent or Asst. Superintendent                    1 (<1%)           0    (0%)          0     (0%)         1     (<1%)
           No Response                                               0 (0%)            1   (<1%)          4     (1%)         5      (1%)
  What grades do you currently teach?
             Elementary and K-8                                    103 (30%)         110 (24%)           85    (18%)       197     (22%)
             Middle School (Grades 6-8)                            178 (52%)         270 (59%)          265    (57%)       515     (57%)
             High School (Grades 9-12)                              29 (9%)           40 (9%)            69    (15%)       108     (12%)
             Substitute Teacher                                      1 (<1%)           1 (<1%)            2    (<1%)         2     (<1%)
             Special Education (Alternative grades only)             3 (1%)            1 (<1%)            4     (1%)         7      (1%)
             Middle and High School grades                           4 (1%)            6 (1%)             5     (1%)        11      (1%)
             None at this time                                       10 (3%)           10 (2%)           14     (3%)        27      (3%)
             No Response                                             13 (4%)           18 (4%)           20     (4%)        42      (5%)
  How many years have you been employed in
  education?
            1st year                                                14     (4%)       23    (5%)         19     (4%)        37       (4%)
            2-3 years                                               28     (8%)       51   (11%)         73    (16%)       109      (12%)
            4-5 years                                               39    (11%)       62   (14%)         59    (13%)       115      (13%)
            6-10 years                                              97    (28%)      106   (23%)        111    (24%)       226      (25%)
            11-20 years                                             96    (28%)      119   (26%)        126    (27%)       247      (27%)
            Over 20 years                                           61    (18%)       84   (18%)         73    (16%)       161      (18%)
            0 or No Response                                         6     (2%)       11    (2%)          3     (1%)        14       (2%)
      Note: Percentages may not total 100% due to rounding error or items in which respondents may respond to all that apply.



      UMass Donahue Institute
      Research and Evaluation Group                                                                                                         81
Annual Report of the MMSP State-level Evaluation                                                          Appendix F



                                                       Year 1      Year 2              Year 3        Total to Date
                           Item
                                                       n   (%)     n     (%)      n       (%)        n       (%)
   Which of the following content areas are you
   currently teaching?
             Mathematics                               190 (56%)   252   (55%)    288      (62%)     535     (59%)
             Elementary (all subjects)                 103 (30%)   107   (23%)     79      (17%)     189     (21%)
             Elementary Mathematics                      N/A        31    (7%)     26       (6%)      45      (5%)
             General Science                            65 (19%)    71   (16%)     56      (12%)     131     (14%)
             Biology                                    22 (6%)     24    (5%)     16       (3%)      41      (5%)
             Physics                                    11 (3%)     16    (4%)     10       (2%)      20      (2%)
             Earth Science                              21 (6%)     16    (4%)     14       (3%)      30      (3%)
             Chemistry                                  12 (4%)     14    (3%)     11       (2%)      22      (2%)
             Any science area*                          88 (26%)    96   (21%)     83      (18%)     183     (20%)
             Technology/Engineering                     10 (3%)      3    (1%)      4       (1%)      10      (1%)
             Other                                      61 (18%)    25    (5%)     22       (5%)      59      (6%)
             Computer Science                            N/A         2   (<1%)      3       (1%)       4     (<1%)
             Do not teach currently                     13 (4%)     10    (2%)     18       (4%)      32      (4%)
   In which type of school do you currently work?
             Public School                             327 (96%)   434   (95%)    445      (96%)     857     (94%)
             Public Charter School                       5 (2%)     14    (3%)      9       (2%)      26      (3%)
             Private School                              8 (2%)      7    (2%)      6       (1%)      20      (2%)
             No Response                                 1 (<1%)     1   (<1%)      4       (1%)       6      (1%)
   Currently hold certification through the National
   Board for Professional Teaching Standards.
             In Any Subject                             51 (15%)        N/A             N/A                N/A
             In Mathematics                              N/A           11 (2%)    14          (3%)   23          (3%)
             In General Science                          N/A            1 (<1%)   3           (1%)    5          (1%)




      UMass Donahue Institute
      Research and Evaluation Group                                                                                 82
Annual Report of the MMSP State-level Evaluation                                                          Appendix F



                          Item                           Year 1        Year 2            Year 3     Total to Date
                                                         n     (%)     n     (%)     n       (%)    n      (%)
    Approximately how many students do you teach
    annually?
                                       0 students          2    (1%)     3    (1%)     3     (1%)     4    (<1%)
                                       1-10 students       6    (2%)     4    (1%)     8     (2%)    16     (2%)
                                       11-40 students     98   (29%)   118   (26%)   101    (22%)   214    (24%)
                                       41-150 students   203   (60%)   295   (65%)   310    (67%)   596    (66%)
                                       151+ students      21    (6%)    28    (6%)    20     (4%)    47     (5%)
                                       No Response        11    (3%)     8    (2%)    22     (5%)    32     (4%)
    Approximately how many students do you teach
    annually who are Title I students?
                                       0 students         66   (19%)   117   (26%)   131    (28%)   222    (24%)
                                       1-10 students      73   (21%)    69   (15%)    66    (14%)   137    (15%)
                                       11-40 students     46   (13%)    58   (13%)    49    (11%)   107    (12%)
                                       41-150 students    39   (11%)    63   (14%)    43     (9%)   107    (12%)
                                       151+ students       6    (2%)     7    (2%)     7     (2%)    14     (2%)
                                       No Response       111   (33%)   142   (31%)   168    (36%)   322    (35%)
    Approximately how many students do you teach
    annually who are academically advanced
    students?
                                       0 students         92   (27%)   154   (34%)   134    (29%)   262    (29%)
                                       1-10 students      61   (18%)    66   (15%)    65    (14%)   129    (14%)
                                       11-40 students     53   (16%)    65   (14%)    80    (17%)   154    (17%)
                                       41-150 students     7    (2%)    14    (3%)    18     (4%)    31     (3%)
                                       151+ students       1   (<1%)     1   (<1%)     0     (0%)     2    (<1%)
                                       No Response       127   (37%)   156   (34%)   167    (36%)   331    (36%)

    Approximately how many students do you teach
    annually who are Special Education students?
                                     0 students           19    (6%)    32    (7%)    26     (6%)    49     (5%)
                                     1-10 students       129   (38%)   144   (32%)   143    (31%)   301    (33%)
                                     11-40 students      123   (36%)   200   (44%)   216    (47%)   398    (44%)
                                     41-150 students      18    (5%)    23    (5%)    17     (4%)    44     (5%)
                                     151+ students         1   (<1%)     0    (0%)     5     (1%)     5     (1%)
                                     No Response          51   (15%)    57   (13%)    57    (12%)   112    (12%)

    Approximately how many students do you teach
    annually who are English Language Learners?
                                    0 students            62   (18%)    88   (19%)    89    (19%)   176    (19%)
                                    1-10 students        112   (33%)   149   (33%)   135    (29%)   262    (29%)
                                    11-40 students        47   (14%)    62   (14%)    78    (17%)   137    (15%)
                                    41-150 students       26    (8%)    36    (8%)    31     (7%)    64     (7%)
                                    151+ students          4    (1%)     5    (1%)     4     (1%)    12     (1%)
                                    No Response           90   (26%)   116   (25%)   127    (27%)   258    (28%)




      UMass Donahue Institute
      Research and Evaluation Group                                                                                 83
Annual Report of the MMSP State-level Evaluation                                                                Appendix F




                                                         Year 1          Year 2             Year 3       Total to Date
                           Item
                                                        n        (%)     n     (%)      n        (%)     n       (%)
   Why did you participate in this course? *
     To increase knowledge in content                    355 (85%)       573 (80%)      499      (72%)   1427    (78%)
     To obtain graduate credit                           278 (66%)       456 (63%)      444      (64%)   1178    (64%)
     To earn PDPs for recertification                    161 (38%)       309 (43%)      240      (35%)    710    (39%)
     To pursue a personal interest                       147 (35%)       245 (34%)      207      (30%)    599    (33%)
     To get an additional license (certification)         89 (21%)       157 (22%)      103      (15%)    349    (19%)
     To prepare for the Massachusetts Test for              82 (20%)     154 (21%)      118      (17%)    354    (19%)
         Educator Licensure (MTEL)
     To earn PDPs for your HOUSSE plan                      53 (13%)     116 (16%)          90   (13%)    259    (14%)
         requirement
     To follow an administrator’s suggestion                32   (8%)        73 (10%)       47   (7%)     152     (8%)
     To obtain a first license (certification)              15   (4%)        22 (3%)        22   (3%)      59     (3%)
     Other                                                  25   (6%)        31 (4%)        34   (5%)      90     (5%)
    No Response                                              2   (1%)         2 (<1%)        0   (0%)       0     (0%)

   High Need District
     Yes                                                 202 (59%)       276 (61%)      254      (55%)    530    (58%)
     No                                                  129(38%)        161 (35%)      199      (43%)    342    (38%)
     Private School (not included)                         8 (2%)          7 (2%)         6       (1%)     20     (2%)
     No Response                                           2 (1%)         12 (3%)         6       (1%)     17     (2%)
   Highly Qualified
     Yes                                                 131 (38%)       267 (59%)      333      (72%)    561    (62%)
     No                                                   92 (27%)        69 (15%)       36       (8%)    120    (13%)
     In some, but not all areas taught                    19 (6%)         21 (5%)        22       (5%)     38     (4%)
     Not enough information to determine                  78 (23%)        82 (18%)       49      (11%)    139    (15%)
     Private School (not included)                         8 (2%)          7 (2%)         6       (1%)     19     (2%)
     Not applicable (not currently teaching)              13 (4%)         10 (2%)        18       (4%)     32     (4%)

   Do you have a High Objective Uniform State Standard of
   Evaluation (HOUSSE) plan? (Public School Teachers Only)
     Yes                                                   75 (23%)      177(40%)       196      (43%)    332    (38%)
     No                                                   142 (43%)      135 (30%)      137      (30%)    285    (32%)
     Not Sure                                              98 (30%)      127(28%)       106      (23%)    237    (27%)
     No Response                                           17 (5%)         9 (2%)        16       (4%)     29     (3%)
   If you do have a HOUSSE plan, how many PDP
   hours do you have in your content area(s)?
      Less than 48 PDP hours                                 9   (12%)       11 (6%)        17    (9%)     32    (10%)
      48 to 100 PDP hours                                    8   (11%)       29 (16%)       19   (10%)     34    (10%)
      101 to 250 PDP hours                                  36   (48%)       58 (33%)       61   (31%)    112    (34%)
      251+ PDP hours                                         8   (11%)       47 (27%)       69   (35%)    107    (32%)
      No Response                                           14   (19%)       32 (18%)       30   (15%)     47    (14%)
          *
          *Data for this item represents the number of seats filled from all courses, rather than from unique
          participants.




      UMass Donahue Institute
      Research and Evaluation Group                                                                                      84
Annual Report of the MMSP State-level Evaluation                                                             Appendix F



                                   Item – Year 1                     Bachelors   Masters   CAGS   Doctoral
                     A degree currently held for each
                     major.
                        Education                                       160      182       10        2
                        Math                                             27        5        0        0
                        General Science                                   8        4        0        0
                        Biology                                          22        3        0        1
                        Chemistry                                         7        3        0        0
                        Earth Science                                     3        1        0        0
                        Physics                                           1        0        0        0
                        All science/technology combined                  41       11        0        1
                        Other                                           106       48        4        1
                     A degree currently being pursued
                     for each major.
                        Education                                         1       34        6        3
                        Math                                              0       55        5        0
                        General Science                                   0        4        0        0
                        Biology                                           0        0        0        0
                        Chemistry                                         0        1        0        0
                        Earth Science                                     0        0        0        0
                        Physics                                           0        0        1        0
                        All science/technology combined                   0        5        1        0
                        Other                                             0        9        4        0

                                   Item – Year 2                     Bachelors   Masters   CAGS   Doctoral
                     A degree currently held for each major.
                        Education*                                       48        65       3        0
                        Math                                             32         4       0        0
                        Math Education*                                  10        11       0        0
                        General Science                                   2         1       0        0
                        Biology                                          29         3       0        0
                        Chemistry                                         3         1       0        1
                        Earth Science                                     3         3       0        0
                        Physics                                           1         3       0        0
                        Technology/Engineering                            0         2       0        0
                        Other Science                                    13        12       0        2
                        All science/technology combined                  51        25       0        3
                     A degree currently being pursued for each major.
                        Education*                                        0        18       3        1
                        Math                                              0        22       1        0
                        Math Education*                                   1        55       3        2
                        General Science                                   0         6       1        0
                        Biology                                           0         0       1        0
                        Chemistry                                         0         0       0        0
                        Earth Science                                     0         0       0        0
                        Physics                                           0         0       0        1
                        Technology/Engineering                            0         0       0        0
                        Other Science                                     0         0       0        0
                        All science/technology combined                   1        10       2        1
*This question only asked of secondary level mathematics teachers.



       UMass Donahue Institute
       Research and Evaluation Group                                                                                 85
Annual Report of the MMSP State-level Evaluation                                                        Appendix F



                                Item – Year 3                   Bachelors   Masters   CAGS   Doctoral
                  A degree currently held for each major.
                     Education                                       162    179        3        2
                     Math                                             46      9        0        0
                     Math Education                                   15     19        0        0
                     Science Education                                 3      9        0        2
                     General Science                                  10      3        0        0
                     Biology                                          19      3        0        1
                     Chemistry                                         9      1        0        0
                     Earth Science                                     6      1        0        0
                     Physics                                           2      1        0        0
                     All science/technology combined                  46      9        0        1
                     Other                                           166     83        4        5
                  A degree currently being pursued for each major.
                     Education                                         3      54       9        1
                     Math                                              2      15       1        0
                     Math Education                                    3      84      14        0
                     Science Education                                 3       8       0        0
                     General Science                                   0       0       0        0
                     Biology                                           0       2       0        0
                     Chemistry                                         0       0       0        0
                     Earth Science                                     0       0       1        0
                     Physics                                           0       0       0        0
                     All science/technology combined                   0       2       1        0
                     Other                                             2      11       2        1




      UMass Donahue Institute
      Research and Evaluation Group                                                                             86
Annual Report of the MMSP State-level Evaluation                                                          Appendix F



                        Item – Total to Date *             Bachelors       Masters   CAGS      Doctoral
             A degree currently held for each
             major.
                Education                                    163             174        1         2
                Math                                          47              11        0         0
                Math Education                                14              19        0         0
                Science Education                              4              10        0         1
                General Science                               10               3        0         0
                Biology                                       22               1        0         0
                Chemistry                                      7               1        0         0
                Earth Science                                  7               2        0         0
                Physics                                        2               1        0         0
                Technology                                   Unknown      Unknown    Unknown   Unknown
                Other Science                                Unknown      Unknown    Unknown   Unknown
                All science/technology combined               48               8        0         0
                Other                                        165              88        1         7
             A degree currently being pursued
             for each major.
                Education                                       3             52        9         2
                Math                                            3             12        2         0
                Math Education                                  2             89       11         0
                Science Education                               2              9        0         0
                General Science                                 0              0        0         0
                Biology                                         0              1        0         0
                Chemistry                                       0              0        0         0
                Earth Science                                   0              0        1         0
                Physics                                         0              0        0         0
                Technology                                   Unknown      Unknown    Unknown   Unknown
                Other Science                                Unknown      Unknown    Unknown   Unknown
                All science/technology combined                 0              1        1         0
                Other                                           3             12        4         1
                   *Based on the last survey completed by each individual subject.




      UMass Donahue Institute
      Research and Evaluation Group                                                                               87
Annual Report of the MMSP State-level Evaluation                                                          Appendix F



                                                             MTEL             MTEL       Scores
                        MTEL tests taken – Year 1            Taken            Passed    Unknown
                        General Curriculum (formerly
                                                             51 15%       46      90%   0       0%
                             Elementary)
                        Mathematics                          23 7%        13      57%   5       22%
                        Middle School Mathematics            43 13%       34      79%   5       12%
                        Middle School Mathematics/
                                                             7     2%      3      43%   2       29%
                             Science
                        General Science                      14    4%     12 86%        0        0%
                        Biology                               7    2%      4 57%        1       14%
                        Chemistry                             2    1%      2 100%       0        0%
                        Physics                               0    0%      0   0%       0        0%
                        Earth Science                         0    0%      0   0%       0        0%
                        Technology/Engineering                2    1%      1 50%        0        0%


                                                             MTEL             MTEL       Scores
                        MTEL tests taken – Year 2            Taken            Passed    Unknown
                        General Curriculum (formerly
                                                             42    9%     40      95%   0       0%
                             Elementary)
                        Elementary Mathematics                4 1%         2      50%   2       50%
                        Mathematics                          41 9%        29      71%   0        0%
                        Middle School Mathematics            82 18%       66      80%   8       10%
                        Middle School Mathematics/
                                                             11    2%      5      45%   0       0%
                             Science
                        General Science                      23 5%        22 96%        0       0%
                        Biology                               4 1%         4 100%       0       0%
                        Chemistry                             0 0%         0   0%       0       0%
                        Physics                               0 0%         0   0%       0       0%
                        Earth Science                         1 <1%        1 100%       0       0%
                        Technology/Engineering                0 0%         0   0%       0       0%



                                                                                            Scores
                     MTEL tests taken – Year 3         MTEL Taken       MTEL Passed
                                                                                         Unknown
                     General Curriculum (formerly
                                                        30        6%     27       90%       0        0%
                          Elementary)
                     Elementary Mathematics              9         3%     8       89%       1     11%
                     Mathematics                        56        12%    49       88%       1      2%
                     Middle School Mathematics         116        25%   102       88%       2      2%
                     Middle School Mathematics/
                                                        15        3%     12       80%       1        7%
                          Science
                     General Science                    18         4%    17       94%       1        6%
                     Biology                             8         2%     6       75%       0        0%
                     Chemistry                           2        <1%     2      100%       0        0%
                     Physics                             0         0%     0        0%       0        0%
                     Earth Science                       1        <1%     1      100%       0        0%
                     Technology/Engineering              0         0%     0        0%       0        0%




      UMass Donahue Institute
      Research and Evaluation Group                                                                               88
Annual Report of the MMSP State-level Evaluation                                                  Appendix F


                                                                                    Scores
                     MTEL tests taken – Total to Date   MTEL Taken   MTEL Passed
                                                                                   Unknown
                     General Curriculum (formerly
                                                        80     9%     73    91%    0         0%
                          Elementary)
                     Elementary Mathematics              15    2%     12    80%    3     20%
                     Mathematics                         92   10%     70    76%    2      2%
                     Middle School Mathematics          190   21%    161    85%    9      5%
                     Middle School Mathematics/
                                                        25     3%     15    60%    4     16%
                          Science
                     General Science                    35     4%     31    89%    1         3%
                     Biology                             8     1%      5    63%    0         0%
                     Chemistry                           3    <1%      3   100%    0         0%
                     Physics                             0     0%      0     0%    0         0%
                     Earth Science                       2    <1%      2   100%    0         0%
                     Technology/Engineering              1    <1%      0     0%    0         0%




      UMass Donahue Institute
      Research and Evaluation Group                                                                       89
Annual Report of the MMSP State-level Evaluation                                                         Appendix F




                                                             Year 1   Year 2          Year 3     Total to Date
                License Areas - Titles as of 10/01
                                                         n   (%)      n   (%)     n      (%)     n     (%)
       01       Academically Advanced                      0 (0%)       2 (<1%)     0     (0%)     1    (<1%)
       02       Biology                                   22 (6%)      34 (7%)     23     (5%)    56     (6%)
       03       Business                                   5 (1%)       3 (1%)      7     (2%)    11     (1%)
       04       Chemistry                                  9 (3%)       6 (1%)      8     (2%)    15     (2%)
       05       Dance                                      0 (0%)       0 (0%)      0     (0%)     0     (0%)
       06       Early Childhood                           17 (5%)      12 (3%)     13     (3%)    24     (3%)
       07       Earth Science                              3 (1%)       4 (1%)      5     (1%)    10     (1%)
       08       Elementary                               203 (60%)    237 (52%)   201    (43%)   439    (48%)
       09       English                                    6 (2%)      14 (3%)      6     (1%)    19     (2%)
       10       ESL                                        5 (1%)       6 (1%)      6     (1%)    10     (1%)
       11       Foreign Language                           2 (1%)       1 (<1%)     3     (1%)     3    (<1%)
       12       General Science                           34 (10%)     57 (13%)    38     (8%)    89    (10%)
       13       Health/Family & Consumer Science           3 (1%)       1 (<1%)     3     (1%)     5     (1%)
       14       History                                    9 (3%)      15 (3%)     14     (3%)    23     (3%)
       15       Instructional Technology                   2 (1%)       4 (1%)      4     (1%)     8     (1%)
       16       Latin and Classic Humanities               0 (0%)       0 (0%)      0     (0%)     0     (0%)
       17       Library                                    0 (0%)       0 (0%)      3     (1%)     3    (<1%)
       18       Mathematics                               45 (13%)     83 (18%)   114    (25%)   192    (21%)
       19       Middle School                             45 (13%)     60 (13%)    48    (10%)   102    (11%)
       20       Middle School Humanities                   0 (0%)       2 (<1%)     2    (<1%)     4    (<1%)
       21       Middle School Mathematics                 43 (13%)     74 (16%)   113    (24%)   183    (20%)
       22       Middle School Math/Science                 2 (1%)      11 (2%)     16     (3%)    22     (2%)
       23       Music                                      0 (0%)       0 (0%)      1    (<1%)     1    (<1%)
       24       Physical Education                         3 (1%)       2 (<1%)     3     (1%)     7     (1%)
       25       Physics                                    4 (1%)       3 (1%)      2    (<1%)     5     (1%)
       26       Political Science/Political Philosophy     0 (0%)       0 (0%)      2    (<1%)     2    (<1%)
       27       Reading                                   17 (5%)      19 (4%)     14     (3%)    25     (3%)
       28       Speech                                     2 (1%)       0 (0%)      0     (0%)     1    (<1%)
       29       Students w/ Moderate Disability           53 (16%)     51 (11%)    82    (18%)   139    (15%)
       30       Students w/ Severe Disability              0 (0%)       0 (0%)      6     (1%)     6     (1%)
       31       Deaf and Hard of Hearing                   0 (0%)       0 (0%)      1    (<1%)     2    (<1%)
       32       Visually Impaired                          0 (0%)       0 (0%)      0     (0%)     0     (0%)
       33       Technology/Engineering                     1 (<1%)      0 (0%)      6     (1%)     4    (<1%)
       34       Theater                                    0 (0%)       1 (<1%)     0     (0%)     0     (0%)
       35       Transitional Bilingual Education           1 (<1%)      5 (1%)      2    (<1%)     7     (1%)
       36       Visual Arts                                4 (1%)       2 (<1%)     0     (0%)     4    (<1%)
      37/38     Other                                     29 (9%)      59 (13%)    29     (6%)    76     (8%)
       39       Elementary Mathematics                      N/A         4 (1%)      9     (2%)    13     (1%)




      UMass Donahue Institute
      Research and Evaluation Group                                                                              90
Annual Report of the MMSP State-level Evaluation                                                                                   Appendix F




                                                     Year 1                Year 2                     Year 3           Total to Date
                         Grade of Certification
                                                     n        (%)          n        (%)       n               (%)      n         (%)
             A        Pre-K – 2                          2 (1%)                1 (<1%)                2       (<1%)         3    (<1%)
             B        Pre-K – 3                          16       (5%)         14     (3%)        12           (3%)        23     (3%)
             C        Pre-K – 6                          36 (10%)              19     (4%)        34           (7%)        68     (7%)
             D        Pre-K – 8                          66 (19%)          102 (22%)              92           (20%)    188       (21%)
             E        Pre-K – 9                          32       (9%)         37     (8%)        30           (6%)        69     (8%)
             F        1–6                            117 (34%)             144 (32%)          120              (26%)    253       (28%)
             G        5–8                                73 (22%)          130 (29%)          175              (38%)    280       (31%)
             H        5–9                                68 (20%)          112 (25%)              78           (17%)    174       (19%)
             I        5 – 12                             37 (11%)              51 (11%)           55           (12%)    112       (12%)
             J        8 – 12                             16       (5%)         35     (8%)        39           (8%)        74     (8%)
             K        9 – 12                             52 (16%)              68 (15%)           91           (20%)    148       (16%)
             L        All Levels                         45 (13%)              52 (11%)           54           (12%)    106       (12%)



                                                          Year 1               Year 2                  Year 3          Total to Date
                            Level of Certification
                                                         n        (%)          n      (%)         n           (%)      n         (%)


             P          Provisional Certification         76       (22%)        86 (19%)          116          (25%)       206     (23%)

                        Initial (Provisional with
             I                                           117       (34%)       181 (40%)          198          (43%)       357     (39%)
                        Advanced Standing)

             S          Standard**                       364 (107%)            500 (110%)         460          (99%)       925    (102%)


             T          Temporary                             4     (1%)            2 (<1%)               0     (0%)         3     (<1%)


          * This reflects the number of teachers teaching science and/or technology courses. The total is not a sum of
          the individual content areas, as many of these teachers teach multiple science courses.
          ** Percentage over 100% due to some participants holding multiple standard licenses.

          Note: Percentages may not total 100% due to rounding error or items in which respondents may respond
          to all that apply




      UMass Donahue Institute
      Research and Evaluation Group                                                                                                        91
Annual Report of the MMSP State-level Evaluation                                                                        Appendix G




  Appendix G: High Need District Eligibility

High Need Districts (See list below.):

1. For proposals with a mathematics content focus: A district is considered to be a high need district if it has a
    mathematics proficiency index for grades 4-8 that is below the state target for Cycle II for MMSP Year 1projects or
    below the state target for Cycle III for MMSP Year 2 projects.. Priority will be given to high need districts with two or
    more schools identified for improvement in mathematics.
2. For proposals with a science and/or technology/engineering content focus: A district is considered to be a high need
    district if it has a science proficiency index for grades 5-8 in 2003 that is at or below the 20th percentile for the state.
In addition, a high need district must demonstrate that there is a high number or percentage of teachers in the district who are
teaching in the academic subject or grade level for which they have not demonstrated subject matter competency through
licensure or completion of the professional development activities in their HOUSE plans.
An interested district that is not identified as high need is encouraged to contact a high need district to explore becoming a
partner in the proposed program (e.g., vocational technical schools are encouraged to contact feeder school districts).
Year 1 High Need Districts
           DISTRICT               MATH       SCIENCE                              DISTRICT                MATH     SCIENCE
                                               T/E                                                                   T/E
AVON                                                                  TAUNTON                                          
BARNSTABLE                                                            WALTHAM                                           
BOSTON                                                               WARE                                             
BROCKTON                                                             WAREHAM                                
CAMBRIDGE                                                            WEBSTER                                           
CHELSEA                                                              WEST SPRINGFIELD                                   
CHICOPEE                                                             WESTFIELD                              
CLARKSBURG                                                                                                             
                                                                       WINCHENDON
EASTHAMPTON                                                           WINTHROP                                           
EVERETT                                                                                                                 
                                                                       WORCESTER
FAIRHAVEN                                                             ABBY KELLEY FOSTER CS                  
FALL RIVER                                        
FITCHBURG                                                            ATLANTIS CS                                       
FLORIDA                                                               BENJAMIN BANNEKER CS                              
GARDNER                                                              BOSTON RENAISSANCE CS                             
GREENFIELD                                                            CONSERVATORY LAB CS                                
HAVERHILL                                                            EDWARD BROOKE CS                                   
HOLBROOK                                                             FREDERICK DOUGLASS CS                             
HOLYOKE                                                              LAWRENCE FAMILY DEV CS                            
HULL                                                                  LOWELL COMMUNITY CS                               
LAWRENCE                                                             NEIGHBORHOOD HOUSE CS                              
LOWELL                                                               NEW BEDFORD GLOBAL CS                             
LYNN                                                                 NEW LEADERSHIP HMCS                               
MALDEN                                                               NORTH CENTRAL REG CS                              
MEDFORD                                                               ROBERT M. HUGHES CS                               
METHUEN                                                               SABIS INTERNATIONAL CS                            
NEW BEDFORD                                                          SEVEN HILLS CS                                    
NORTH ADAMS                                                          SOMERVILLE CS                                     
PITTSFIELD                                                           UPHAMS CORNER CS                                   
PROVINCETOWN                                                          ATHOL-ROYALSTON RSD                               
RANDOLPH                                                             BERKSHIRE HILLS RSD                    
REVERE                                                                FRONTIER RSD                                       
SALEM                                                                GILL-MONTAGUE RSD                      
SOMERVILLE                                                           HAMPSHIRE RSD                                     
SOUTHBRIDGE                                                          HAWLEMONT RSD                          
SPRINGFIELD                                                          MOUNT GREYLOCK RSD                                 
                                                                       RALPH C MAHAR RSD                                 




       UMass Donahue Institute
       Research and Evaluation Group                                                                                               92
Annual Report of the MMSP State-level Evaluation                                                           Appendix G


Year 2 High Need Districts
          DISTRICT               Grade 5    Grade 8 Grades 4-            DISTRICT             Grade 5   Grade 8 Grades 4-
                                      SCI    SCI     8 Math                                    SCI       SCI     8 Math

Attleboro                             √                         Southbridge                     √         √         √
Avon                                          √                 Springfield                     √         √         √
Boston                                √       √         √       Taunton                         √         √
Bourne                                √                         Waltham                                   √
Brockton                              √       √         √       Ware                                      √
Cambridge                             √       √         √       Wareham                                             √
                                                                Webster                         √                   √
Chelsea                               √       √         √
                                                                Westfield                                           √
Chicopee                              √       √         √
                                                                Winchendon                                √         √
Clarksburg                            √       √         √
                                                                Winthrop                        √
Dedham                                        √                 Worcester                       √         √         √
Dracut                                        √                 Academy Of Strategic CS                             √
East Bridgewater                      √                         Benjamin Banneker CS            √         √         √
Easthampton                           √                         Frederick Douglass CS                     √         √
Everett                               √       √                 Edward Brooke CS                √
Fairhaven                                     √                 Conservatory Lab CS             √
Fall River                            √       √         √       Community Day CS                √
Fitchburg                             √       √         √       Sabis International CS          √         √
Florida                                       √                 Neighborhood House CS           √         √
Gardner                                       √         √       Abby Kelley Foster Reg CS       √
                                                                So.Boston Harbor Academy CS     √
Greenfield                                              √
                                                                Robert M. Hughes Academy CS     √         √
Haverhill                             √                 √
                                                                Lawrence Family Dev. CS         √         √         √
Holbrook                                      √
                                                                Lowell Community CS             √                   √
Holyoke                               √       √         √       New Leadership HMCS                       √         √
Hull                                  √       √                 New Bedford Global HMCS                   √         √
Lawrence                              √       √         √       North Central CS                          √         √
Leominster                            √                         Boston Renaissance CS           √         √         √
Lowell                                √       √         √       Seven Hills CS                  √         √         √
Lynn                                  √       √         √       Somerville CS                                       √
Malden                                √       √         √       Prospect Hill Academy CS        √
Medford                               √       √                 Uphams Corner CS                √
Methuen                               √       √                 Atlantis CS                     √         √         √
New Bedford                           √       √         √       Adams-Cheshire                  √
                                                                Athol-Royalston                 √         √
North Adams                           √       √         √
                                                                Berkshire                                           √
Orange                                √
                                                                Freetown-Lakeville              √
Oxford                                √
                                                                Gateway                         √
Quincy                                √                         Gill-Montague                   √
Pittsfield                                    √         √       Hampshire                                 √         √
Randolph                                      √                 Hawlemont                                           √
Revere                                √       √                 New Salem-Wendell               √
Rockland                                      √
Salem                                 √       √         √
Somerville                            √       √         √




      UMass Donahue Institute
      Research and Evaluation Group                                                                                     93
Annual Report of the MMSP State-level Evaluation                                                              Appendix G


Year 3 High Need Districts
        DISTRICT          Grade 5      Grade 8   Grades 4-8           DISTRICT               Grade 5    Grade 8   Grades 4-8
                         SCIENCE      SCIENCE      MATH                                     SCIENCE    SCIENCE      MATH
    Attleboro                √                                Waltham                                     √
    Avon                                 √                    Ware                                        √
    Boston                   √           √           √        Wareham                                                 √
    Bourne                   √                                Webster                          √                      √
    Brockton                 √           √           √        Westfield                                               √
    Cambridge                √           √           √        Winchendon                                  √           √
    Chelsea                  √           √           √        Winthrop                         √
    Chicopee                 √           √           √        Worcester                        √          √           √
    Clarksburg               √           √           √        Academy Of Strategic CS                                 √
    Dedham                               √                    Benjamin Banneker CS             √          √           √
    Dracut                               √                    Frederick Douglass CS                       √           √
    East Bridgewater         √                                Edward Brooke CS                 √
    Easthampton              √                                Conservatory Lab CS              √
    Everett                  √           √                    Community Day CS                 √
    Fairhaven                            √                    Sabis International CS           √          √
    Fall River               √           √           √        Neighborhood House CS            √          √
    Fitchburg                √           √           √        Abby Kelley Foster Regional      √
    Florida                              √                    CS
    Gardner                              √           √        South Boston Harbor              √
    Greenfield                                       √        Academy CS
                                                              Robert M. Hughes Academy         √          √
    Haverhill                √                       √        CS
    Holbrook                             √                    Lawrence Family                  √          √           √
    Holyoke                  √           √           √        Development CS CS
                                                              Lowell Community                 √                      √
    Hull                     √           √                    New Leadership HMCS                         √           √
    Lawrence                 √           √           √        New Bedford Global HMCS                     √           √
    Leominster               √                                North Central CS                            √           √
    Lowell                   √           √           √        Boston Renaissance CS            √          √           √
    Lynn                     √           √           √        Seven Hills CS                   √          √           √
    Malden                   √           √           √        Somerville CS                                           √
    Medford                  √           √                    Prospect Hill Academy CS         √
    Methuen                  √           √                    Uphams Corner CS                 √
    New Bedford              √           √           √        Atlantis CS                      √          √           √
    North Adams              √           √           √        Adams-Cheshire RSD               √
    Orange                   √                                Athol-Royalston RSD              √          √
    Oxford                   √                                Berkshire Hills RSD                                     √
    Quincy                   √                                Freetown-Lakeville RSD           √
    Pittsfield                           √           √        Gateway RSD                      √
    Randolph                             √                    Gill-Montague RSD                √
    Revere                   √           √                    Hampshire RSD                               √           √
    Rockland                             √                    Hawlemont RSD                                           √
    Salem                    √           √           √        New Salem-Wendell RSD            √
    Somerville               √           √           √
    Southbridge              √           √           √
    Springfield              √           √           √
    Taunton                  √           √




      UMass Donahue Institute
      Research and Evaluation Group                                                                                       94
Annual Report of the MMSP State-level Evaluation                                                                                 Appendix H




  Appendix H: Subject Matter Competency Demonstration Options


How do teachers demonstrate subject matter competency in Massachusetts?


  Elementary teachers                                                    Middle and secondary school teachers
  may demonstrate competence in reading, writing,                        may demonstrate subject matter competence in
  mathematics, and other areas of the basic                              each of the areas they are teaching through one of
  elementary school curriculum through one of the                        the following:
  following:

                                                                         Passing the Massachusetts Test for Educator
  Passing the Massachusetts Test for Educator                            Licensure (MTEL) appropriate Subject Matter Test;
  Licensure (MTEL) Elementary Subject Matter Test:                         Middle School Humanities
    General Curriculum and the                                             Middle School Mathematics
      Foundations of Reading                                               Middle School Mathematics/Science
                                                                           Subject Title (e.g., History, English, Physics)


                                                                         Making sufficient progress* on Massachusetts High
                                                                         Objective Uniform State Standard of Evaluation
                                                                         (HOUSSE)
  `Making sufficient progress* on Massachusetts High
  Objective Uniform State Standard of Evaluation
  (HOUSSE)                                                               Completion of an appropriate academic major
                                                                         Completion of an appropriate graduate degree
                                                                         Completion of comparable coursework equivalent to
                                                                         an undergraduate academic major
                                                                         Advanced certification or credentialing
  *It is the Department's expectation that a teacher will have … completed at least half [48] of the content PDPs [96 total] needed to meet
  HOUSSE requirements before being considered highly qualified.




Charter School Teachers who teach core academic subjects do not need a Massachusetts license,
but must hold a Bachelor's degree and demonstrate competence in the subject area in which they
teach. Charter school teachers may demonstrate subject matter competence through any one of the
options available to elementary and middle/secondary teachers.

Teachers in Vocational Schools who teach core academic courses are required to meet the
definition of a highly qualified teacher. A vocational school teacher who teaches a core academic
subject must hold a Bachelor's degree, be licensed or certified by the state, and demonstrate subject
matter competence in order to be considered highly qualified.
(information obtained from MADOE, 2004)




        UMass Donahue Institute
        Research and Evaluation Group                                                                                                     95
Appendix I: Highly Qualified Status of Unique Participants


    Highly Qualified Status of Unique Participants

                                           Took only one course                                                                                         Took multiple courses
                                                 n = 555                                                                                                      n = 354
                                               Y2                       Y3                                                              Y2 only                    Y3 only           Multiple Years*




                                                                                                   Only one




                                                                                                                                                                                                                        Multiple
                                                                                       Subtotal




                                                                                                                                                                                                            Subtotal


                                                                                                                                                                                                                                   courses
                                                                                                              course
                          Y1                 n = 196                  n = 223                                          Y1 only          n = 31                     n = 54                   n = 248

                        n=136         C1               C2      C1               C2                                      n=21     C1               C2         C1              C2      C1               C2                                     TOTAL
    STATUS                           n=185             n=11   n=209             n=14                                             n=25             n=6       n=35             n=19   n=241             n=7                                     N=909
    Became Highly
                          5           6                 0      8                 0                19                     1        2               1          4                3      81               0                92                     111
    Qualified
    Highly Qualified
    in some content       4           6                 0      7                 0                17                     2        2               0          1                0      14               0                19                      36
    areas but not all
    Highly Qualified
    but unable to
                         52          107                8     145                9                321                    9       14               2          21              13     104               3                166                    487
    determine when
    became HQ
    Not Highly
                         34           27                2      20                0                83                     4        2               0          3                0      13               1                23                     106
    Qualified
    Private school or
                         11           8                 1      11                3                34                     2        2               1          0                2      9                1                17                      51
    not teaching

    Unknown              30           31                0      18                2                81                     3        3               2          6                1      20               2                37                     118

     *Participants who participated in courses from partnerships in both cohorts 1 and 2 are included in the cohort 1 column as they initially began
     their participation in MMSP in cohort 1.




     UMass Donahue Institute
     Research and Evaluation Group                                                                                                                                                                                                                    96
Appendix J: Criteria that Account for Gain in Highly Qualified Status

    Criteria that Account for Gain in Highly Qualified Status
    All unique participants to date
                                   Gained
    Criteria Accounting for                    Gained HQ status      Gained HQ status         TOTAL
                                  HQ status
    Gains in Highly                             during Year 2         during Year 3       became HQ over
                                   during
    Qualified Status                                                                      course of project
                                   Year 1     Cohort 1   Cohort 2   Cohort 1   Cohort 2
    MTEL only                          1          9         0          5          1               16
    National Certification only      NA           0         0          0          0                0
    Degree in Content Area
                                       1          0         0          3          1                5
    only
    Undergrad Equivalent only        NA           0         1          2          0                3
    HOUSSE only                       10         43         0         17          0               70
    Earned a Teaching License          0          1         0          2          0                3
    MTEL/Undergrad Equiv
                                     NA           1         0          3          0                4
    combined
    MTEL/Degree combined               0          0         0          2          0                2
    MTEL/HOUSSE combined               0          5         0          0          0                5
    HOUSSE/Undergrad Equiv
                                       0          0         0          1          1                2
    Combined
    MTEL/HOUSSE/Undergrad
                                     NA           0         0          1          0                1
    Equiv combined
    TOTAL                             12         59         1         36          3              111




     UMass Donahue Institute
     Research and Evaluation Group                                                                            97

								
To top