Docstoc

Evaluating the policy impact of PISA Task Force Fostering

Document Sample
Evaluating the policy impact of PISA Task Force Fostering Powered By Docstoc
					1
    Organisation for Economic Cooperation and Development (OECD)




        Potential impact of PISA


                        Michael Davidson
                    OECD/Directorate for Education
2                          What is PISA?
    A three-yearly assessment that…
      … examines the performance of 15-year-olds in key
          subject areas as well as a wider range of
          educational outcomes
           • Including students attitudes to learning, their beliefs
             about themselves, and their learning strategies
      … collects contextual data
        from students schools, parents and systems to
        identify policy levers
              Contextual data explain 71% of the performance variation
               among schools internationally
    Coverage
         Representative samples of between 3,500 and
          50,000 15-year-old students drawn in each country
         Most federal countries also draw regional samples
          3
          3 1st results from TALIS




                                             Results from PISA 2006
          Learning Environments
Creating Effective Teaching and




                                        Scientific knowledge and use of that
                                         knowledge in…
                                         … identifying scientific issues,
                                         … explaining scientific phenomena,
                                         … using scientific evidence
4    PISA provides key benchmarks for
      the quality of education systems
    1. Overall performance of education systems
    2. Equity in the distribution of learning
       opportunities
         Measured by the impact students’ and schools’
          socio-economic background has on performance…
        … not merely by the distribution of learning
          outcomes
    3. Consistency of performance standards
       across schools
    4. Gender differences
5                      Mean science scores
                  300     350          400           450       500         550          600

            Finland
            Canada
              Japan
      New Zealand
          Australia
       Netherlands
              Korea
           Germany
                 UK
    Czech Republic
        Switzerland                                                             The centre
            Austria                                                              boundaries
                                                                            Theline is the
            Belgium
            Ireland                                                         indicate the 95%
                                                                                mean
           Hungary             There is only a                              confidence limits
            Sweden             significant
             Poland
           Denmark             difference if there
             France            is no overlap of
            Croatia            scores
            Iceland
     United States
    Slovak Republic
              Spain
            Norway
        Luxembourg
               Italy
           Portugal
             Greece
             Serbia
           Bulgaria
             Turkey
           Romania
        Montenegro
            Mexico


                         OECD (2007), PISA 2006 – Science Competencies for Tomorrow’s World, Table 2.1c
                                                                                                                                                                                         6



                                                                                                                                           100
                                                                                                                                                      120
                                                                                                                                                            140




                                                                                                         0
                                                                                                             20
                                                                                                                  40
                                                                                                                        60
                                                                                                                                 80
                                                                                             Germany
                                                                                       Czech Republic
                                                                                              Austria
                                                                                             Hungary
                                                                                          Netherlands
                                                                                              Belgium
                                                                                                Japan
                                                                                                Italy
                                                                                               Greece
                                                                                       Slovak Republic
                                                                                               Turkey
                                                                                          Switzerland
                                                                                                Korea
                                                                                          Luxembourg
                                                                                        United States
                                                                                             Portugal
                                                                                               Mexico
                                                                                       United Kingdom
                                                                                         New Zealand
                                                                                             Australia
                                                                                               Canada
                                                                                                                       across schools
                                                                                                                                                                                             Is it all innate ability?




                                                                                              Ireland
                                                                                                                                                                  Variation in student performance




                                                                                             Denmark
                                                                                                Spain
                                                                                               Poland
                                                                                              Sweden
                                                                                              Norway
                                                                                                                                  Performance variation




                                                                                              Iceland
                                                                                               Finland
OECD (2007), Learning for tomorrow’s world: First results from PISA 2006, Table 4.1a
                                                                                         -80
                                                                                               -60
                                                                                                     -40
                                                                                                           -20
                                                                                                                                         100
                                                                                                                                               120




                                                                                                                 0
                                                                                                                     20
                                                                                                                          40
                                                                                                                               60
                                                                                                                                    80
                                                                             Bulgaria
                                                                            Germany
                                                                      Czech Republic
                                                                              Austria
                                                                            Hungary
                                                                         Netherlands
                                                                              Belgium
                                                                                Japan
                                                                                 Italy
                                                                              Greece
                                                                     Slovak Republic
                                                                              Turkey
                                                                         Switzerland
                                                                             Romania
                                                                               Serbia
                                                                              Croatia
                                                                                Korea
                                                                         Luxembourg
                                                                       United States
                                                                             Portugal
                                                                              Mexico
                                                                                   UK
                                                                         Montenegro
                                                                        New Zealand
                                                                           Australia
                                                                              Canada
                                                                              Ireland
                                                                            Denmark
                                                                                Spain
                                                                               Poland
                                                                              Sweden
                                                                             Norway
                                                                              Iceland
                                                                                                                                                     7 Variation in student performance




PISA 2006: Science Competencies for Tomorrow’s World, Figure 4.1a.




                                                                              Finland
                                                                                         -80
                                                                                               -60
                                                                                                          -40
                                                                                                                     -20
                                                                                                                                                         100
                                                                                                                                                                   120




                                                                                                                           0
                                                                                                                               20
                                                                                                                                    40
                                                                                                                                         60
                                                                                                                                              80
                                                                             Bulgaria
                                                                            Germany
                                                                      Czech Republic
                                                                              Austria
                                                                            Hungary
                                                                         Netherlands
                                                                                                                                                   schools



                                                                              Belgium
                                                                                                                                                 Variation of




                                                                                Japan
                                                                                                                                              performance within




                                                                                 Italy
                                                                              Greece
                                                                     Slovak Republic
                                                                              Turkey
                                                                         Switzerland
                                                                             Romania
                                                                               Serbia
                                                                              Croatia
                                                                                Korea
                                                                         Luxembourg
                                                                       United States
                                                                             Portugal
                                                                              Mexico
                                                                                   UK
                                                                                                     schools




                                                                         Montenegro
                                                                                                   Variation of




                                                                        New Zealand
                                                                           Australia
                                                                                               performance between




                                                                              Canada
                                                                              Ireland
                                                                            Denmark
                                                                                Spain
                                                                               Poland
                                                                              Sweden
                                                                             Norway
                                                                              Iceland
                                                                                                                                                                         8 Variation in student performance




PISA 2006: Science Competencies for Tomorrow’s World, Figure 4.1a.




                                                                              Finland
     High science performance
                   565
                       Finland
9                                   Average performance
                                    of 15-year-olds in
                       545
                                    science – extrapolate
                                    and apply
                       Hong Kong-China

    Chinese Taipei  Canada
          Estonia   Japan
                     New Zealand
                     Australia
                    525
                    Netherlands
     Liechtenstein  Korea


       Equity
                     Slovenia
    United Kingdom Germany
    Czech Republic Switzerland
      Macao-China Austria
           Ireland  Belgium
                    505
                    Hungary
                    Sweden
                    Poland
             France Denmark
           Iceland Croatia
    United States Latvia
                    Slovak Republic, Spain, Lithuania
            Norway 485
                    Luxembourg
                       Russian Federation
          Portugal     Italy
                       Greece

                       465



                       Israel

                       445
          16         … 18 countries perform below this line   6

     Low science performance
10
         Impact of selected student and school factors
11                  on school performance
               (after accounting for all other factors in the model)
     %
 100
  90
  80
                        Schools with greater
  70                    autonomy (resources)
  60                     School activities to
  50                    One additional learning
                      promote sciencehour of
                       self-study or homework
  40                    One additional hour of
                      science learning at school
  30                    School results posted
  20                           publicly
                        Academically selective
                       schools but no system-
  10                        wide effect
   0                  Schools practising ability
                      grouping for all subjects
  10
  20                                               Effect after accounting
  30                                                for the socio-economic
                                                   background of students,
                                                          Net
                                                     schools and countries

                      OECD (2007), PISA 2006 – Science Competencies from Tomorrow’s World, Table 6.1a
Creating Effective Teaching and
                                           1st results from TALIS
          Learning Environments                                     12
                                                                    12



                                               PISA
                                  Evaluating the policy impact of
13   External evaluation of the policy impact of PISA


      Objective
          To what degree does PISA impact policies,
           meet expectations and priorities of the
           participants?
      Evaluation   criteria
          Relevance
          Effectiveness
          Sustainability
          Unexpected / Unplanned impacts
14   External evaluation of the policy impact of PISA
      Team of three external evaluators
            David Hopkins, Dianne Pennock, Jo Ritzen
        Two-step procedure
            Questionnaire to stakeholders in the country
            In-depth evaluation of 5 case studies
        Stakeholder groups
            Policy makers at the national/federal and local level
             as well as members of the PISA Governing Board
            School principals, representatives of teacher and
             parent organisations
            Academics and researchers
            Representatives of the business community and the
             media
        Evaluation report
            Conclusions and recommendations based on the
             empirical results obtained
15   Strategy for an evaluation of the policy impact
                        of PISA
        Effectiveness – Specific questions
            To what extent are the participating countries and other
             stakeholders aware of the output results of PISA?
            To what extent are these output results being used as input
             into the policy-making processes of clients and other
             stakeholders?
            How significant is the influence of the PISA output results
             used in the policy-making processes?
            What is the role of PISA in the context of national
             assessment strategies, i.e. to what extent have countries used
             PISA to add to, or complement, their national assessments or
             data?
            To what extent has PISA provided added value to the national
             assessment strategies?
            What impact has PISA had on policy coherence in participating
             countries?
16
     Outcomes – Relevance of PISA
     • Policy makers are considered the most
       significant stakeholder group in relation to
       PISA and its results.
     • Policy makers are also considered as the
       major group responsible for implementing
       policies in light of PISA.
     • The various stakeholder groups assume
       relatively low levels of responsibility for the
       PISA results in their countries.
17
     Outcomes – Relevance of PISA
     • Overall, PISA is seen as a useful tool for
       identifying and addressing nationally relevant
       themes and problems
     • In many countries PISA is used to monitor
       and evaluate the quality and equity of the
       education system
     • Impact of PISA varies from country to
       country:
       • In countries with unsatisfactory outcomes
          often direct policy impact
       • In high-performing systems: PISA as an
          evaluation mechanism
18
             Important aspects of PISA
               Relative importance of the aspects of PISA results
                                     (2.1)
                            All countries (n=659)
             Ranking

       School context

     Home background

            Attitudes

             Science

               Math

             Reading

                        0%   10%   20%      30%   40%     50%    60%     70%   80%     90%   100%
          Extremely and Very   Moderately    Not very and Not at all   Don't know   No response
19      Outcomes – Effectiveness and
           sustainability of PISA
     • In all countries, PISA is seen as an important
       instrument for policy making with a high level
       of credibility
     • Impact of PISA on policy formation
       increasing from PISA 2000 to PISA 2006
     • Influence of PISA greater at the national /
       federal level than at the local level or in
       school practices and classroom instruction
     • Policy makers, academics and researchers and
       the media most aware of PISA
20
                   Addressing policy needs
               Effectiveness of PISA in adressing policy needs (2.2)
                           Total (n=644)
            Business community (n=33)
            Teacher associations (n=36)
          Media representatives (n=48)
     Academics and researchers (n=125)
                         Parents (n=65)
                School principals (n=79)
       Local government officials (n=72)
                  Policy makers (n=145)
          National co-ordinators (n=41)

                                           0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
           Extremely and Very    Moderately    Not very and Not at all   Don't know   No response
21
                     Contributing to change
                Success of PISA in contributing to change through
                   the implementation of reforms, policies or
                             initiatives – All countries
                           Total (n=659)
             Business community (n=33)
             Teacher associations (n=36)
          Media representatives (n=49)
     Academics and researchers (n=128)
                         Parents (n=69)
                School principals (n=81)
       Local government officials (n=72)
                  Policy makers (n=149)
          National co-ordinators (n=42)

                                           0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
           Extremely and Very    Moderately    Not very and Not at all   Don't know   No response
22

               Effectiveness of PISA in complementing and adding
                   value to national assessment strategy (2.4)
                           Total (n=644)
            Business community (n=33)
            Teacher associations (n=36)
          Media representatives (n=48)
     Academics and researchers (n=125)
                         Parents (n=65)
                School principals (n=79)
       Local government officials (n=72)
                  Policy makers (n=145)
          National co-ordinators (n=41)

                                           0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
           Extremely and Very    Moderately    Not very and Not at all   Don't know   No response
23      Outcomes – Effectiveness and
           sustainability of PISA
     • According to respondents, the influence of
       PISA could be increased by:
       •    a better coordinated and strategic approach for
           the dissemination of PISA results;
       •    further support for various stakeholder groups in
           interpreting PISA results and in designing
           strategies for improvement in light of PISA;
       •    In some countries, clearer linkage of PISA
           results to national or federal assessment
           strategies; and,
       •    greater utilisation of the PISA results by
           participating countries.
24
         Unexpected / unplanned outcomes

     •    Great public interest and debate
     •    PISA as an instrument to justify reform
     •    “Culture of blame”
     •    Increased interest in empirical educational
          research
25
                                         Next step
                                      Evaluation criteria
                               Unexpected/Unplanned Impacts & Sustainability
                                                Committee
                                                Orientation*

                                                 Expected                                                 Impacts (3)




                                                                                                                        Member countries
                                                                                                                        Policy impacts in
                                                Outcomes (3)
        Policy needs,
            issues        Relevance                                        Effectiveness
              and                                Expected
                                                                                                          Impacts (2)
       problems facing                          Outcomes (2)
           Member
          countries
                                                 Expected
                                                                                                          Impacts (1)
                                                Outcomes (1)



                                                                        Committee functioning

                                                                                                Output
                                                               Inputs         Processes
                                                                                                Results
     *Also includes any objectives defined in
      committee mandates. Level 3 Expected
      Outcomes include the OECD’s
      Strategic Objectives.
                                                                            Efficiency
26



     Thank you very much for your attention!

            http://www.pisa.oecd.org

           Michael.davidson@oecd.org

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:8
posted:10/3/2012
language:English
pages:26