EVALUATION OF PHASE 1 OF THE

                                   Summary Report

The Schools for Innovation and Excellence (SIE) initiative is one of four initiatives in
the Victorian Government’s Innovation and Excellence in the Middle Years program,
developed and funded to provide all schools with the opportunity to develop
innovative approaches and programs in the middle years (Years 5-9). The other three
initiatives are:
    Middle Years Reform Program (ongoing)
    Restart (concluded December 2004)
    Access to Excellence (concluded December 2005).
SIE commenced in 2003 and provides support for primary, secondary and specialist
schools to work together in clusters to raise levels of innovation and excellence in
Victorian schools. By 2005, all Victorian government schools were involved in a
cluster, and a total of 247 clusters were operating. It is anticipated that school clusters
will implement a range of activities that will enhance teaching and learning programs,
bring about significant school organisational reform and partnerships with local
SIE built on the work of the Middle Years Research and Development Project
(MYRAD) in which 260 schools worked in 62 clusters to improve learning
opportunities for students in the middle years of schooling. Findings from the
MYRAD study identified the advantages of clustering. Subsequently, DE&T has
encouraged schools to co-operate in developing ways to improve learning
opportunities for students, and thus improve outcomes in the middle years of
schooling. These outcomes include student participation and engagement, and
transition as well as achievement in literacy and numeracy.
In September 2002, schools were invited to join Phase 1 of SIE. These schools
indicated their willingness to work in clusters with a common set of understandings
about teaching and learning, framed by the model for school improvement expressed
in The General Design for a Whole-School Approach to School Improvement (Hill
and Crévola, 1997). This approach provides the conceptual framework within which
SIE can be implemented in the context of a whole school plan. Many schools in
Victoria use the model and incorporate strategies for improvement in their existing
school plans.
Phase 1 of the SIE project commenced in 2003. It includes 70 clusters and will extend
to 2007. Phase 2 commenced in 2004, includes 90 clusters and will finish in 2007.
Phase 3 commenced in 2005, includes 87 clusters and will finish in 2007.
Clusters received first-year grants averaging $200,000 to support cluster projects,
including the funding of a cluster Educator. The appointment of a cluster Educator
was not compulsory in the first year, but the equivalent of one full time teacher was
required to support the innovation. In their subsequent years of operation, clusters are

allocated a further $80,000 to employ a cluster Educator. It was anticipated that
clusters would develop and implement a range of proposals to improve teaching and
learning programs for students in the middle years, to bring about significant school
organisational reforms, and to strengthen partnerships with the local school

In May 2005 the Australian Council for Educational Research (ACER) was
commissioned to conduct an evaluation of Phase 1 of SIE involving approximately
490 schools in 70 clusters. The objectives of the evaluation are evident in the six key
questions that were addressed:
   1. What is the impact of SIE in Phase 1 schools on:
          literacy and numeracy achievement levels of students in Years 5-8
          participation rates of these students
          engagement rates of these students
          transition between primary and secondary schools?
   2. How has clustering as a model contributed to the achievement of the
      initiative’s objectives?
   3. To what extent do the findings confirm research concerning the relationship
      between achievement, engagement and transition?
   4. What are the facilitating and inhibiting factors in the success of the cluster
   5. What other effects, either benefits or disadvantages have resulted from the SIE
      Phase 1 initiative?
   6. What advice, in the light of an analysis of the responses to questions 1-5, can
      be provided on the value of this model or variations to the model, as a vehicle
      for achieving Government objectives for middle years education in the future?
The evaluation proceeded through several steps.

Step 1: Analysis of existing student data for Phase 1 and Phase 3 schools
Initially, the first and third evaluation questions were addressed by means of a
multivariate, multilevel analysis of existing data about student achievement and social
outcomes. The extent to which valid answers can be provided to these questions was
crucially dependent on the availability of such outcome measures at the student-level.
That is, to evaluate the impact of the clustering initiative, the available AIM data for
student literacy (Reading) and numeracy (Mathematics) achievements and
engagement levels since 2002 in Phase 1 schools was analysed, together with
comparable data from Phase 3 schools.
Phase 3 schools constituted a suitable comparison group as they did not participate in
SIE in 2003. Phase 1 schools were regarded as a ‘treatment’ group. Any changes over
time in student outcomes can be contrasted with the ‘control’ group – Phase 3
schools. These contrasts can also be investigated between clusters. As the schools

were not randomly allocated to ‘treatment’ and ‘non-treatment’ groups, it was
important to take into account the characteristics of the schools, including, for
example, their average socio-economic status, size, whether single sex or co-
educational, whether the secondary school in a cluster covers Years 7 to 10, 7 to 12,
or 11 to 12, school size and so on.
A key question addressed in the analysis was:
    To what extent has clustering ‘added-value’ to students’ literacy and numeracy
     achievements given: (a) their prior achievement (as measured by AIM Reading
     and AIM Mathematics); and (b) their ‘intake’ characteristics (i.e., gender,
     language background, family socio-economic status, Indigenous status, and
     school SLN index)?
Step 1 thus opened up possibilities for the exploration of relationships between
student achievement and student attitudes, such as engagement and transition, to be

Step 2: The survey component of the evaluation
The survey component of the evaluation built on Step 1 and was implemented in
Term 3, 2005. Special questionnaire items and scales were developed to gather data
about the ways in which clustering was implemented in Phase 1 schools and to test
relationships between these patterns of implementation and:
    the value-added measures of outcomes from Step 1 (data files were merged for
     this purpose);
    judgements about the level of impact made by principals, cluster coordinators,
     teachers closely involved in the program and regional staff.
Data were collected from all schools that participated in Phase 1. Four survey
instruments were developed:
      questionnaire for cluster Educators
      questionnaire for principals
      questionnaire for primary teachers
      questionnaire for secondary teachers.
The questionnaires addressed the following evaluation questions:
   1. To what extent did Phase 1 contribute to the achievement of the objectives of
      the SIE initiative:
           improved literacy and numeracy among students of Years 5-8
           improved participation and engagement of these students
           improved transition from primary to secondary school?
   2. What aspects of clustering, as a model, contributed to the achievement of the
      initiative’s objectives? (Sub-questions about clustering asked for further details
      about the strategies used by schools to promote collaboration and to implement
      changes; for example, to teaching, curriculum and assessment, and school

   3. Which features of cluster leadership action contributed to the effectiveness of
   4. Which factors facilitated or impeded the effectiveness of the cluster model to
      achieve SIE objectives?

Step 3 Analysis of survey data
Three different analyses were conducted. The comparative study involved a value-
added analysis of AIM data from Phase 1 and Phase 3 schools. The main purpose of
this study was to compare the 2004 literacy and numeracy achievements of Year 7
students in Phase 1 clusters with the achievements of Year 7 students in Phase 3
clusters. The comparison took account of the prior achievement of the students in
2002 when they were in Year 5. The analyses also took student gender, language
background and Indigenous status into account.
The second analysis focused on the survey data, and described the main features
identified by principals, cluster Educators and teachers in the questionnaires.
Descriptive results from the survey were grouped around a number of themes,
including: the cluster initiative, cluster identity, impact on schools, impact on teaching
and teachers, impact on students, and the sustainability of the changes.
Thirdly, further analysis described differences between clusters that were lower than
expected, expected and higher than expected student achievement in AIM
mathematics. A path analysis was undertaken to investigate the relative importance of
various factors in producing the outcomes of the SIE initiative.

Step 4: Reporting
On completion of the three analyses, a report was written to explain the findings
identified by the different analyses.

   Analyses of AIM data for Reading and Mathematics indicated that, overall,
     students in Phase 1 clusters made relatively better progress than students in
     Phase 3 clusters in Mathematics but not in Reading from Year 5 in 2002 to
     year 7 in 2004.
   It may be too early to draw conclusions from AIM data about the impact of
     SIE, given that Phase 1 clusters commenced operation in 2003 and the
     quantitative evaluation was based on data from the 2004 AIM testing, after
     only a single year of cluster activity. The impact of SIE on student learning can
     be expected to be broader than the Reading and Mathematics skills measured
     by the AIM tests.
   Students in Phase 1 schools were reported to be more engaged in learning, and,
     to a minor or moderate extent, to demonstrate improving levels of literacy and
     numeracy, making connections across the learning domains and making more
     effective use of ICT.

    Survey responses indicate a clear recognition that improving student learning
     outcomes is directly linked to the provision of professional learning for
     teachers. Around 90% of schools used SIE funding for professional learning
     within schools. Cluster Educators played a leadership role in the organisation
     of professional learning activities directly relevant to local needs and interests.
    Support provided for cluster Educators was an important element in the
     operation of clusters, including networking with other cluster Educators, and
     the support of principals and school leadership teams. School leadership teams
     were reported to have provided strong leadership support for clusters.
    Changes reported as having taken place in schools as a result of participation in
     SIE included changes in teaching practices, school organisation and practices
     affecting transition from primary to secondary school.
    Survey responses provided evidence of a range of outcomes for teachers and
     teaching, with the strongest agreement being in relation to teachers catering
     more effectively for the diversity of individual needs and learning styles in
     Years 5-9, an increase in collaboration between primary and secondary
     schools, increased teacher understanding of the differences between primary
     and secondary schools, and the implementation of more effective strategies to
     support transition.
    The identified changes in teaching and teachers’ understanding and practices,
     enabling teachers to provide better learning opportunities for students, are
     likely to lead to improved student outcomes in the key areas targeted by SIE
     (literacy, numeracy, engagement, participation and transition).
    It was found that, typically, the effect of clustering was not direct, but mediated
     through various aspects of the school, with school leadership in particular
     being a consistently important factor.

   That further monitoring of improvements in AIM reading and mathematics be
    undertaken in schools over time, and that the performance of the Year 6 cohort
    of students in Phase 1 cluster schools be studied when they are in Year 9 in
   That clusters collect and report on a broad range of data about student
    performance in literacy and numeracy, including performance against the
    Victorian Essential Learning Standards for English and Mathematics.
   That the leadership role of cluster Educators be recognised in future
    appointments to the role, and that the vital role of principals and school
    leadership teams in facilitating effective clusters be acknowledged.
   That a focussed evaluation of the range of school-level and cluster-level
    professional learning activities be undertaken, with a view to identifying key
    features of teacher professional learning in the context of a reform initiative
    such as SIE.

    That further monitoring of the impact of cluster initiatives in Phase 1 SIE
     schools be conducted during 2006, when it will be possible to track changes
     over the four years from 2003-2006.
    That professional learning for school leaders and leadership teams be provided
     as further support for SIE.
    That additional data be collected and analyses conducted to further investigate
     the outcomes for Phase 1 schools after a longer period of implementation of
    That a longitudinal approach to evaluating the impact of SIE be adopted. Such
     an evaluation approach should be planned for the state level, the regional level
     and the cluster level, and should be incorporated into all planning, ideally from
     the commencement of the initiative.
    That each region and each cluster map out goals, the criteria of success for
     each goal and how each goal is to be evaluated, and establish a process to chart
     their own trajectories of change over time.

Case studies of six Phase 1 clusters were prepared by Dr Brian Sharpley. These
clusters were identified by regional office middle years staff, and performance in
relation to AIM data and teacher judgement data.
The case studies suggest that effective clusters have the following characteristics:
       the cluster is ‘natural’ - primary and secondary schools, and specialist
        schools if applicable, are located reasonably close geographically
       a ‘community of learners’ is established with staff, students, parents and the
        broader community
       principals regularly communicate with each other, have a strong sense of
        working for a common purpose, have developed a collective vision and
        have reduced competition between each other and their schools
       schools within the cluster share data and collaborate on strategies to act on
        their data
       staff across the cluster visit each others’ schools for purposeful observation,
        collaborative meetings, etc.
       there is a collegiate approach to teaching across the cluster (teachers team
        teach, share ideas, shadow and coach each other)
       teachers are aware of the changing nature of the profession, adopt
        initiatives and are comfortable with advice from the central office
       there is a strong reliance on a high-performing cluster coordinator.

Key messages for clusters
   The importance of the formulation of a clear and shared cluster vision - each
     case study cluster demonstrated a clearly articulated vision.

    The importance of a practical - and practicable – implementation strategy -
     each case study cluster recognised the need for change, collected data,
     identified common issues, and formulated a plan.
    The importance of strong support from principal and other school leaders
    The importance of the employment of an effective cluster educator/coordinator
     - a visionary prepared to take risks, able to identify the change levers and
     skilled in galvanising communal support for change.
    The importance of teacher ownership - active involvement in developing the
     cluster vision, decision-making regarding implementation, design and delivery
     of cluster professional learning, and participation in action research.
    The importance of providing time for teachers from across the cluster to
     engage in discussion and reflection, including visits to other schools within and
     beyond the cluster, and shadowing and mentoring opportunities.
    The importance of restructuring organisational practices – considering
     common meeting times, professional learning, cluster team meetings,
     reorganised timetables.

These         case        studies        can        be            accessed          at


To top