Docstoc

NAPLAN_report_2012_modelling_participation_testing

Document Sample
NAPLAN_report_2012_modelling_participation_testing Powered By Docstoc
					Modelling the effects of
participation on achievement
in NAPLAN testing
Ray Adams

September 2012
PROFESSOR RAY ADAMS—ABOUT THE AUTHOR

Professor Adams is a Professorial Fellow of the University of Melbourne and an independent
consultant who specialises in psychometrics, educational statistics, large-scale testing and
international comparative studies.

He has been the International Project Director for the OECD Programme for International
Student Achievement (PISA) since its inception. PISA is perhaps the world's largest and most
significant educational research project.

Professor Adams has a PhD in educational measurement and educational statistics from the
University of Chicago and has published widely on the technical aspects of educational
measurement. His item response modelling software packages (Quest and ACER ConQuest) are
amongst the most widely used in educational and psychological measurement.

In addition to his directorship of PISA he has acted as a consultant to numerous large-scale
assessment projects both in Australia and internationally. He has served as chair of the
technical advisory committee for the International Association for the Evaluation of Educational
Achievement (IEA) and as Head of Measurement at the Australian Council for Educational
Research (ACER).

Professor Adams is currently chair of the expert advisory group guiding the development of
Australian National Literacy and Numeracy tests, and he is chair of the ACER Psychometrics
Institute.




                                                                                               ii
Modelling the effects of participation on
   achievement in NAPLAN testing




                  Ray Adams
                September 2012
Contents
1.   Introduction ..................................................................................................................................... 3
2.   Participation categories ................................................................................................................... 3
3.   Patterns of participation .................................................................................................................. 4
4.   A model for the impact of non-participation................................................................................... 5
5.   Estimated impact of non-participation ............................................................................................ 9
6.   The impact of imputation .............................................................................................................. 11
7.   Summary ....................................................................................................................................... 14
Appendix 1:         Percentage of students present, Numeracy by Year, Year level and Jurisdiction ......... 16
Appendix 2:         Percentage of students exempt, Numeracy by Year, Year level and Jurisdiction ......... 17
Appendix 3:         Percentage of students withdrawn, Numeracy by Year, Year level and Jurisdiction ... 18




                                                                                                                                                       2
1. Introduction
This report describes an exploratory analysis of the relationship between participation and
achievement in NAPLAN at a jurisdictional level. High participation in testing gives the most
accurate picture of achievement. Except for the Northern Territory, participation rates are high in
every jurisdiction, with more than 90% of students participating. However, differences in participation
rates may affect achievement if the ability of participating and non-participating students differs.

Since 2008, NAPLAN data have been collected in May of each year and are available for analysis
from around October in the same year. The data provide achievement level information for all
students in Australia at Years 3, 5, 7 and 9 in the areas of Numeracy, Reading, Writing, Spelling and
Grammar and Punctuation. In addition to achievement, the NAPLAN database includes information
for each student that indicates their gender, language background, Indigenous status, geographic
location, parent’s level of education, parent’s occupation and school.

2. Participation categories
All students within the Year 3, 5, 7 and 9 cohorts are classified according to their NAPLAN
participation status. This classification occurs separately for each of the tests.

   Exempt students are identified by schools according to specified guidelines. Exemptions may be
    due to disability or, for recent international arrivals, limited English language proficiency. Exempt
    students are not given a NAPLAN score, but are included in the count of students that are
    estimated to be working at or below national minimum standard. They are not included in the
    calculation of mean scores.
   Absent students are students who did not sit the test because they were absent from school on the
    day that the test was administered and were unable to sit the follow-up test. Imputation techniques
    are used to provide NAPLAN scores for these students and they are included in all reporting.
   Withdrawn students are students that did not sit the test because they were withdrawn from the
    testing program by their parent/carer. The reasons why parents (or carers) withdraw a student vary
    widely. Imputation techniques are used to provide NAPLAN scores for these students and they
    are included in all reporting.
   Sanctioned abandonment occurs when a student responds to one or more questions but is then
    forced to abandon the test due to illness, injury or misadventure and is unable to sit the follow-up
    test. The item response data from sanctioned abandonment students is discarded and then
    imputation techniques are used to provide NAPLAN scores for these students and they are
    included in all reporting.
   Present students are students that attend the testing session and are not sanctioned abandonments.
    These students are given a NAPLAN score and are included in all reporting.

In summary, all students are included in estimates of the percentage of students working at or below
minimum standards and all students, with the exception of exempt students, are included in all other
aspects of the reporting. For absent, withdrawn and sanctioned abandonment students imputation
techniques are used to produce the NAPLAN scores (see section 6). The retention of these students in
the database, made possible through imputation, both simplifies the structure of the database and
reduces, but does not eliminate, the potential for non-response bias. The imputation process uses the
student variables: Indigenous status, language background other than English (LBOTE), geolocation,
parental occupation, parental education, school and gender.


                                                                                                           3
3. Patterns of participation

PRESENT

Appendix 1 shows the percentage of Present students broken down by Year level, time and
Jurisdiction. The results for Australia overall are also presented. For the purposes of illustration the
participation information is based upon the numeracy test, the results for the other assessment
domains are very similar.

Perhaps the three most striking features of the results are first the decline in the Present rate from
2008 to 2009 to 2010 at all Year levels and for all jurisdictions with the exception of the Northern
Territory. For 2010 to 2011 the Present rate is stable at the national level for all Year levels, although
there is variation between jurisdictions – for example in the primary Years the Present rates for South
Australia and the Australian Capital Territory decline, while that for Victoria increases. The general
pattern of decline from 2008 to 2010/2011 is due to the increasing number of exempt and/or
withdrawn students. Second, the pattern across the larger jurisdictions is reasonably consistent with
New South Wales typically having the highest rate of present students and Victoria amongst the
lowest. Third, there was a substantial increase in the Present rate for the Northern Territory between
2008 and 2009, with subsequent declines in 2010 and 2011.


EXEMPT

Appendix 2 shows the percentage of Exempt students broken down by Year level, time and
Jurisdiction. The results for Australia overall are also presented. For the purposes of illustration the
participation information is based upon the numeracy test, the results for the other assessment
domains are very similar.

The percentage of Exempt students has increased at all Year levels and for all jurisdictions with the
exception of South Australia (which began from a high base in 2008, but has been increasing since
2009). Further, the pattern across the larger jurisdictions is reasonably consistent with New South
Wales and Western Australia typically having the lowest rate of exempt students and Victoria
amongst the highest.


WITHDRAWN

Appendix 3 shows the percentage of Withdrawn students broken down by Year level, time and
Jurisdiction. The results for Australia overall are also presented. For the purposes of illustration the
participation information is based upon the numeracy test, the results for the other assessment
domains are very similar.

The percentage of Withdrawn students has increased at all Year levels and for all jurisdictions. South
Australia has the largest percentage of Withdrawn students, with its rate typically being at least twice
the national average.


ABSENT

Although not shown here, the overall percentage of Absent students has remained relatively consistent
over time but the rate does increase with Year level. At Year 3 the rate is typically just over 3%,


                                                                                                           4
whereas at Year 9 it is 6.5% to 7%. For Years 3 and 5 and amongst the larger states Victoria and
Western Australia typically have the highest rate of Absent students and New South Wales and
Queensland the lowest. For Years 7 and 9 and amongst the larger states Victoria and Tasmania
typically have the highest rate of Absent students and New South Wales and Queensland the lowest.

4. A model for the impact of non-participation
The impact that non-participation will have on NAPLAN results is a function of three things:

   the proportion of students that participate;
   the relationship between students’ NAPLAN scores and their participation status; and,
   the strength of the model that is used to impute non-respondent scores from their background
    characteristics – Indigenous, LBOTE, geolocation, parental occupation, parental education, school
    and gender.

In this section a simple hypothetical model of the relationship between participation and NAPLAN
performance is posited and then this model is used to explore the potential impact of participation
rates and participation rate variability on NAPLAN results. Note that the modelling discussed here
does not take into account the corrections that are achieved through the missing data imputations. This
matter is further discussed below.

We begin by considering two random variables, N, which is a student’s NAPLAN proficiency and P,
which is a student’s propensity to participate in NAPLAN. Let us then assume that these two variables
follow a bivariate normal distribution and for simplicity of presentation that each variable is
marginally          If the correlation between the two variables is r, then the bivariate normal density
function is given by:

                                                                                                     (1)

For the case r=0, the distribution is illustrated in Figure 1. In this case there is no relationship between
these two variables.




                     Figure 1:       Illustration of a Bivariate Normal Distribution




                                                                                                           5
An alternative view of this distribution is shown in Figure 2. The top left panel is a top view of the
distribution with zero correlation between NAPLAN and Participation propensity. In addition to
showing the joint distribution the figure also shows the marginal distributions of NAPLAN and
Participation propensity. The top right panel is a top view of the distribution with a correlation of one
between NAPLAN and Participation propensity. That is, the two values are in complete agreement.
The bottom panel shows the case were the two variables are positively correlated, that is higher
NAPLAN values occur with higher Participation propensity values and similarly lower NAPLAN
values occur with lower Participation propensity values, but the values are not in exact agreement.




    Figure 2:       Illustration of Bivariate Normal Distributions and Marginal Distributions with
                                          Differing Correlations




                                                                                                            6
To model the impact of non-participation we further posit that only those students with higher
Propensity to participate values actually sit the NAPLAN test. That is, rather than having access to the
NAPLAN scores of all students we only have access to the NAPLAN scores for those students with a
Propensity to participate that is greater than say, k. This outcome is illustrated in Figure 3.




          Figure 3:       Illustration of the Impact of Censoring with Differing Correlations

In the top right panel we see that the propensity distribution is truncated at k and the cloud of values is
subsequently censored at a propensity of k. There is however, no impact on the NAPLAN distribution
because the NAPLAN and Participation propensity are unrelated. In the top right panel the censoring
has trimmed the line of points at the bottom left and it has also cut (or truncated) the tail of the
NAPLAN distribution, this has occurred because the two sets of values are in complete agreement –
the students with lowest propensity are also those with the lowest NAPLAN scores and they are now
unobserved. This truncation in the NAPLAN distribution will result in an increase in the NAPLAN

                                                                                                          7
mean. The bottom panel of Figure 3 illustrates what happens if the two variables are positively
correlated, but the correlation is not one. In this case the truncation of the Propensity distribution leads
to a loss of values from the left tail NAPLAN distribution with the loss being greatest in the extremes,
but not a complete loss as was the case with a correlation of one. The result is a skew in the
distribution where the magnitude of the skew is a function of the correlation. Similarly the mean of
the distribution will be increased as a function of the correlation.

For the standardised bivariate normal distribution given by (1), the expected value of N is zero. That is
the average NAPLAN score is set at zero. Using (1) and assuming that the NAPLAN distribution is
(partially) censored because students below some level k, for the related variable Propensity
distribution do not have NAPLAN scores, we can estimate the potential impact on the NAPLAN
mean score. In fact the mean of the NAPLAN distributions assuming the censoring is given by:

                                                                                                       (2)

Where        is the standard normal density function and          is the standard normal distribution
function.       , therefore, is the proportion of students that participate in NAPLAN and           is the
ordinate of the density function at k.

As             equation (2) also gives the bias, in standardised scores, that is caused by the level of
non-participation. Note that (2) shows that if r=0, then there is no bias, further if all students
participate then             and             so again, there is no bias. The bias is a function of both r
and k; and finally, if r<0 the bias is negative and if r>0 it is positive.

In practice the value of k can be easily estimated by                  where q is the proportion of
students that participate in NAPLAN. The correlation, r, is more difficult to estimate, and in fact may
well be very hard to estimate precisely. In the results reported below rather than estimating r, a range
of possible r values is used to illustrate the bias values that might reasonably expected as a result of
non-participation. The bias, in this case, is the amount by which the results would be artificially
inflated.

Under this model we can also derive an expression for the difference between the mean scores of the
participants and the non-participants. Using the fact that         we have




                                                                                                       (3)

The non-participation will not only affect the mean, it will also affect other statistics such as the
variance, percentiles and proportion estimated to be working at or below national minimum standard.
The variance, which is reduced by censoring, can be shown to be:

                                                                                                      (4)

The impact on the percentiles, and the proportion estimated to be working at or below national
minimum standard has yet to be derived, but the impact can be easily simulated.


                                                                                                             8
5. Estimated impact of non-participation
Table 1 shows the estimated bias in NAPLAN means for a variety of participation rates and
correlations between Propensity to participate and NAPLAN scores. The values in the table assume a
standard deviation for NAPLAN scores of 70 points.

In practice NAPLAN participation rates, in the larger states, range from about 90% (in Victoria) to
about 95% in New South Wales. The Northern Territory rate is typically lower at about 80-85%. It is
hard to know what a reasonable value for the correlation between Propensity to participate and
NAPLAN score would be, but given the wide variety of reasons for which students do not participate
in NAPLAN the correlation is probably in the range between 0.05 and 0.50, and likely to be nearer the
lower end of this range rather than the upper end. This suggests a bias in the mean from about 0.38 to
6.82 NAPLAN points in a context where the reported confidence intervals for the mean are reported
to be of the order of ±3 NAPLAN points (Table 1).

Given the particular interest in the impact that changes in participation rates might have on trend
indicators or the impact that differences between jurisdiction rates might have on inter-jurisdiction
comparisons it is important to consider not only the absolute bias, but to also compare the bias across
rows in Table 1. For example if the 90% participation rate row is compared to the 95% participation
rate row the differences in the mean range from as little as 0.3 (r=0.05) to just over 3 points (r=0.50).
The actual values are probably closer to the bottom end of this range, so something around 1 point.
For the percentage at or below minimum standard the figures differ by up to about 0.7 percentage
points.

The potential estimated differences between the mean scores of participants and non-participants are
shown in Table 2. In the participation rate range from 90% to 95% the differences vary quite widely –
6.82 to 75.99 points in the range of values that we have been using.

To illustrate the potential impact on the estimated percentage at or below the national minimum
standard we consider a scenario where there are actually 5% of students at or below national
minimum standard. Then, reading from Table 3 if the participation rate was 0.95 and the correlation
between Propensity to participate and NAPLAN score was 0.05 then the estimated percentage at or
below the national minimum standard would be 4.94%, or 0.06 percentage points too low. Similarly
if the participation rate was 0.90 and the correlation between Propensity to participate and NAPLAN
was 0.50 then the estimated percentage would be 3.46%, or 1.54 percentage points too low.

Table 3 shows the estimated proportion of students that would be deemed to be at or below national
minimum standard for combinations of participation rates and the correlation of propensity to
participate and NAPLAN score. The proportions are calculated as if the actual percentage was 5% and
so are differences from this figure. Table 1 and Table 2 have been computed analytically using (2) and
(3), whereas Table 3 has been constructed via simulation and as a consequence there will be small
amounts of sampling variability in the Table 3 figures.




                                                                                                            9
Table 1:          Estimated Bias in the NAPLAN Estimated Mean Score for a Variety of Participation
                  Rates and Correlations between NAPLAN Mean Score and Propensity to Participate

                              Correlation between Propensity to participate and NAPLAN Mean Score
Participation
                0.00   0.05     0.10    0.15    0.20     0.25    0.30     0.35    0.40     0.45      0.50    1.00
    Rate
    0.85        0.00   0.96     1.92    2.88    3.84     4.80    5.76     6.72    7.68     8.64      9.60   19.20
    0.86        0.00   0.91     1.81    2.72    3.62     4.53    5.44     6.34    7.25     8.15      9.06   18.12
    0.87        0.00   0.85     1.70    2.55    3.40     4.26    5.11     5.96    6.81     7.66      8.51   17.02
    0.88        0.00   0.80     1.59    2.39    3.18     3.98    4.77     5.57    6.36     7.16      7.96   15.91
    0.89        0.00   0.74     1.48    2.22    2.96     3.70    4.44     5.18    5.92     6.66      7.39   14.79
    0.90        0.00   0.68     1.36    2.05    2.73     3.41    4.09     4.78    5.46     6.14      6.82   13.65
    0.91        0.00   0.62     1.25    1.87    2.50     3.12    3.75     4.37    5.00     5.62      6.25   12.49
    0.92        0.00   0.57     1.13    1.70    2.26     2.83    3.39     3.96    4.52     5.09      5.66   11.31
    0.93        0.00   0.51     1.01    1.52    2.02     2.53    3.03     3.54    4.04     4.55      5.05   10.11
    0.94        0.00   0.44     0.89    1.33    1.77     2.22    2.66     3.10    3.55     3.99      4.44    8.87
    0.95        0.00   0.38     0.76    1.14    1.52     1.90    2.28     2.66    3.04     3.42      3.80    7.60
    0.96        0.00   0.31     0.63    0.94    1.26     1.57    1.89     2.20    2.51     2.83      3.14    6.28
    0.97        0.00   0.25     0.49    0.74    0.98     1.23    1.47     1.72    1.96     2.21      2.46    4.91
    0.98        0.00   0.17     0.35    0.52    0.69     0.86    1.04     1.21    1.38     1.56      1.73    3.46
    0.99        0.00   0.09     0.19    0.28    0.38     0.47    0.57     0.66    0.75     0.85      0.94    1.88


Table 2:          Estimated Difference in the NAPLAN Mean Score for participants and non-
                  participants for a Variety of Participation Rates and Correlations between NAPLAN
                  Score and Propensity to Participate

                              Correlation between Propensity to participate and NAPLAN Mean Score
Participation
                0.00   0.05    0.10     0.15    0.20    0.25     0.30    0.35    0.40     0.45      0.50     1.00
    Rate
    0.85        0.00   6.40    12.80   19.20    25.60   32.00   38.40    44.80   51.20   57.60      64.00   128.01
    0.86        0.00   6.47    12.94   19.41    25.88   32.35   38.82    45.29   51.76   58.23      64.70   129.41
    0.87        0.00   6.55    13.09   19.64    26.19   32.73   39.28    45.83   52.37   58.92      65.46   130.93
    0.88        0.00   6.63    13.26   19.89    26.52   33.15   39.78    46.41   53.04   59.67      66.30   132.60
    0.89        0.00   6.72    13.44   20.17    26.89   33.61   40.33    47.06   53.78   60.50      67.22   134.45
    0.90        0.00   6.82    13.65   20.47    27.30   34.12   40.95    47.77   54.60   61.42      68.25   136.50
    0.91        0.00   6.94    13.88   20.82    27.76   34.70   41.64    48.58   55.52   62.46      69.40   138.80
    0.92        0.00   7.07    14.14   21.21    28.28   35.35   42.42    49.49   56.56   63.63      70.70   141.39
    0.93        0.00   7.22    14.44   21.66    28.87   36.09   43.31    50.53   57.75   64.97      72.19   144.37
    0.94        0.00   7.39    14.78   22.18    29.57   36.96   44.35    51.75   59.14   66.53      73.92   147.85
    0.95        0.00   7.60    15.20   22.80    30.40   38.00   45.60    53.20   60.80   68.40      75.99   151.99
    0.96        0.00   7.85    15.71   23.56    31.42   39.27   47.13    54.98   62.84   70.69      78.54   157.09
    0.97        0.00   8.18    16.37   24.55    32.73   40.92   49.10    57.29   65.47   73.65      81.84   163.67
    0.98        0.00   8.65    17.29   25.94    34.58   43.23   51.88    60.52   69.17   77.81      86.46   172.92
    0.99        0.00   9.42    18.84   28.27    37.69   47.11   56.53    65.96   75.38   84.80      94.22   188.45




                                                                                                               10
Table 3:          Estimated proportion at or below a cut-point that should yield 5% for a variety of
                  Participation Rates and correlations between NAPLAN Mean Score and Propensity to
                  Participate

                              Correlation between Propensity to participate and NAPLAN Mean Score
Participation
                0.00   0.05     0.10    0.15    0.20     0.25    0.30     0.35    0.40     0.45     0.50   1.00
    Rate
    0.85        5.00   4.87     4.69    4.53    4.30     4.15    3.92     3.64    3.53     3.27     2.96   19.20
    0.86        5.00   4.86     4.77    4.55    4.35     4.18    3.97     3.76    3.56     3.38     2.98   18.12
    0.87        5.00   4.85     4.81    4.60    4.37     4.25    4.02     3.79    3.56     3.38     3.14   17.02
    0.88        5.00   4.93     4.74    4.56    4.44     4.33    4.15     3.85    3.66     3.46     3.22   15.91
    0.89        5.00   4.90     4.79    4.63    4.53     4.37    4.08     3.93    3.76     3.51     3.35   14.79
    0.90        5.00   4.95     4.81    4.64    4.52     4.32    4.22     3.94    3.85     3.59     3.46   13.65
    0.91        5.00   4.86     4.88    4.60    4.63     4.36    4.32     4.06    3.91     3.59     3.53   12.49
    0.92        5.00   4.94     4.96    4.79    4.56     4.44    4.25     4.16    4.01     3.80     3.59   11.31
    0.93        5.00   4.83     4.80    4.66    4.62     4.56    4.38     4.23    4.13     3.86     3.72   10.11
    0.94        5.00   4.82     4.88    4.71    4.71     4.57    4.55     4.27    4.15     3.99     3.87   8.87
    0.95        5.00   4.94     4.89    4.87    4.82     4.58    4.57     4.41    4.30     4.12     4.04   7.60
    0.96        5.00   4.97     4.99    4.73    4.74     4.69    4.61     4.45    4.38     4.24     4.13   6.28
    0.97        5.00   4.94     4.87    4.88    4.81     4.66    4.67     4.58    4.49     4.37     4.40   4.91
    0.98        5.00   4.95     4.90    4.93    4.89     4.81    4.76     4.68    4.60     4.47     4.43   3.46
    0.99        5.00   5.03     4.94    4.94    4.89     4.91    4.94     4.84    4.75     4.77     4.73   1.88



6. The impact of imputation
As mentioned above the NAPLAN scaling does not simply ignore the non-participating students as
has been assumed in the above modelling and calculations. For absent, withdrawn and sanctioned
abandonment students imputation techniques are used to produce the NAPLAN scores. The retention
of these students in the database is made possible through imputation which both simplifies the
structure of the database and reduces, but does not eliminate, the potential for non-response bias.

The imputation process utilises the student demographic variables: Indigenous status, language
background other than English (LBOTE), geolocation, parental occupation, parental education, school
and gender. The consequence is that if these variables account for the correlation between non-
participation and NAPLAN performance then all of the effects described above will be removed from
the results. At the other extreme, if they do not account for any of the correlation then they will have
no impact on the non-participation bias.

Figure 4 shows the mean numeracy scores for New South Wales for Present, Absent and Withdrawn
students, by Year level from 2008 to 2011 – recall that the Absent and Withdrawn students did not
respond to any test items and their proficiency values are imputed based upon their background
characteristics and the estimated relationship between those background characteristics and
achievement. The same information for Victoria is shown in Figure 5 and Figure 6 shows a scatterplot
of the New South Wales and Victorian results.

Figure 4 and Figure 5 show that the Present students are estimated to have the highest mean and
Absent students the lowest. This indicates that Absent students typically have background
characteristics that are associated with lower levels of performance (amongst the assessed students).


                                                                                                              11
For New South Wales (Figure 4) the differences in the Present and Absent means vary from 19 points
for Year 3 in 2008 to 39 points for Year 9 in 2011. For Victoria (Figure 5) the differences vary from 8
points for Year 3 in 2008 to 23 points for Year 7 in 2008. Referring back to Table 2, and assuming the
imputation model accounts for the majority of the non-response bias these differences suggest
correlations between propensity to participate and NAPLAN performance of 0.05 to 0.20, since it is
this range of correlations that suggest differences that have the order of magnitude that are being
observed.

     600




     550




     500




     450




     400




     350
             Present Absent W'drwn Present Absent W'drwn Present Absent W'drwn Present Absent W'drwn
               Y3      Y3     Y3     Y5     Y5     Y5      Y7     Y7     Y7      Y9     Y9     Y9
      2008    409      391   401    488    467     477    552    521     546    593    564     571
      2009    406      383   401    502    478     495    550    518     543    599    564     583
      2010    402      379   395    499    476     491    551    518     539    594    555     578
      2011    406      386   405    500    479     492    550    517     534    592    553     577



Figure 4:           Mean Numeracy Scores for New South Wales for Present, Absent and Withdrawn
                    Students, by Year level, by 2008 to 2011




                                                                                                       12
     600



     550



     500



     450



     400



     350
            Present Absent W'drwn Present Absent W'drwn Present Absent W'drwn Present Absent W'drwn
              Y3      Y3     Y3     Y5     Y5     Y5      Y7     Y7     Y7      Y9     Y9     Y9
     2008    417      409   400    490    480     481    553    531     530    592    570     574
     2009    411      400   408    497    485     490    550    530     547    598    582     599
     2010    411      399   404    503    493     499    555    533     542    595    575     585
     2011    413      400   409    500    487     495    552    529     546    592    568     587



Figure 5:          Mean Numeracy Scores for Victoria for Present, Absent and Withdrawn Students, by
                   Year level, by 2008 to 2011




                                                                                                      13
The scatterplot in Figure 6 highlights that the ordering of mean performance of Present, Withdrawn,
then Absent is consistent across the two jurisdictions. It also shows that for Present students the New
South Wales and Victorian mean are very similar, yet the imputed values for Absent students are
consistently higher for Victoria than New South Wales. Recall that in Victoria the absentee rate is
about 4% for Year 3, 5 and 7 and it is about 8% at Year 9. For New South Wales the absentee rate is
just above 2% at Years 3 and 5, about 3% at Year 7 and just over 5% at Year 9.

                    650


                    600


                    550
    Victoria Mean




                                                                                 Present
                    500
                                                                                 Absent
                                                                                 Withdrawn
                    450
                                                                                 Identity line

                    400


                    350
                          350    400     450    500     550    600     650
                                        New South Wales Mean



Figure 6:                       Scatterplot of Mean Numeracy Scores for Victoria and New South Wales for Present,
                                Absent and Withdrawn students, 2008 to 2011

7. Summary
There are systematic patterns in the participation data that should be addressed. First, whilst the vast
majority of students are present for the NAPLAN assessment there is a clear pattern of decrease in the
number of students who are being assessed, which is explained by increases in the percentage of
students that are either exempt or withdrawn. The decline is not apparent between 2010 and 2011 so
perhaps the decline has reached a plateau. Second, there are systematic patterns in the variation in the
participation rate across jurisdictions; these patterns are reasonably stable over Year levels and time.
Victoria has the lowest participation rate (about 90%) and New South Wales the highest (about 95%).

It is recommended that further work be done on both explaining these observations and examining
their impact on the comparability of results, both across jurisdictions and over time. Mathematical
modelling can be undertaken to examine the degree of threat to comparability. As part of this review,
the extent to which the imputation methodology reduces the impact of non-response bias might also
be considered.

To assist in examining the threat that the observed patterns in participation present to the
interpretation of trend indicators and to inter-jurisdictional comparisons we have posited a flexible
model for the relationship between participation in NAPLAN and NAPLAN achievement. This model
can be used to estimate the potential bias in NAPLAN means and estimates of the proportion of
students working at or below the national minimum standard as a function of the participation rate.


                                                                                                               14
The results reported of the modelling suggest that further exploration of the impact of non-
participation and the functioning of the missing data imputation model is warranted. Following this
cursory review it seems that the posited model for non-participation bias may have some merit and it
suggests that, at the current rates, the impact of non-participation bias on trend indicators is probably
small to negligible. But there may be a substantially important impact of variability in participation
rates on some between jurisdiction comparisons.

The impact of the patterns in the participation data on the validity of trend indicators needs to be
further examined, they are potentially of concern.

The role that the imputation model plays in ameliorating the effects of non-participation would also
seem worthy of further investigation. The NAPLAN imputation approach is statistically quite
complex and the impact that it may or may not have on the indicators is not immediately clear.




                                                                                                        15
Appendix 1:                                              Percentage of students present, Numeracy by Year,
                                                         Year level and Jurisdiction
                                          97                                                                                 97
                                          96                                                                                 96
                                          95                                                                                 95
  Percent Present Grade 3




                                                                                     Percent Present Grade 5
                                          94                                                                                 94
                                          93                                                                                 93
                                          92                                                                                 92
                                          91                                                                                 91
                                          90                                                                                 90
                                          89                                                                                 89
                                          88                                                                                 88
                                          87                                                                                 87
                                          86                                                                                 86
                                          85                                                                                 85
                                          84                                                                                 84
                                                 2008     2009     2010     2011                                                    2008     2009     2010     2011
                                          ACT    92.92    92.12    92.51    90.88                                            ACT    94.40    93.87    92.62    92.11
                                          NSW 95.99       95.81    95.27    95.46                                            NSW 96.41       96.26    95.98    95.86
                                          NT     81.45    90.59    85.70    85.08                                            NT     83.50    92.50    86.47    85.53
                                          QLD    95.30    94.97    93.68    93.98                                            QLD    95.96    95.32    94.50    94.19
                                          SA     93.71    93.02    91.48    90.83                                            SA     94.46    93.81    92.86    92.67
                                          TAS    95.52    95.81    94.28    94.47                                            TAS    95.33    95.59    95.01    94.76
                                          VIC    93.17    91.48    90.67    91.64                                            VIC    93.75    92.21    91.71    92.37
                                          WA     94.09    94.75    93.63    93.92                                            WA     94.69    95.36    94.63    94.32
                                          AUS    94.57    94.16    93.17    93.51                                            AUS    95.14    94.72    94.09    94.07




                                           97                                                                                 97
                                           96                                                                                 96
                                           95                                                                                 95
                                                                                                   Percent Present Grade 9
                Percent Present Grade 7




                                           94                                                                                 94
                                           93                                                                                 93
                                           92                                                                                 92
                                           91                                                                                 91
                                           90                                                                                 90
                                           89                                                                                 89
                                           88                                                                                 88
                                           87                                                                                 87
                                           86                                                                                 86
                                           85                                                                                 85
                                           84                                                                                 84
                                                  2008     2009     2010     2011                                                    2008     2009     2010     2011
                                           ACT    93.62    93.11    93.71    93.38                                            ACT    91.90    89.94    89.38    90.07
                                           NSW 95.59       95.88    95.42    95.30                                            NSW 93.10       93.50    92.94    92.87
                                           NT     78.16    91.06    85.64    84.53                                            NT     76.31    85.16    80.30    82.43
                                           QLD    95.73    95.32    94.71    94.39                                            QLD    92.75    92.53    91.51    90.80
                                           SA     94.37    94.12    92.80    92.86                                            SA     91.02    89.83    87.89    88.43
                                           TAS    94.35    94.07    93.32    92.38                                            TAS    90.00    90.35    88.55    88.02
                                           VIC    93.90    92.81    92.40    92.80                                            VIC    90.19    89.44    88.29    88.85
                                           WA     94.32    94.76    94.37    94.28                                            WA     91.81    93.04    91.70    91.76
                                           AUS    94.71    94.63    94.06    94.00                                            AUS    91.76    91.77    90.72    90.71



                                                                                                                                                                        16
Appendix 2:                             Percentage of students exempt, Numeracy by Year,
                                        Year level and Jurisdiction
                           3.5                                                         3.5

                           3.0                                                         3.0
  Percent Exempt Grade 3




                                                              Percent Exempt Grade 5
                           2.5                                                         2.5

                           2.0                                                         2.0

                           1.5                                                         1.5

                           1.0                                                         1.0

                           0.5                                                         0.5
                                 2008    2009   2010   2011                                  2008   2009   2010   2011
                           ACT   2.08    2.85   2.17   2.13                            ACT   1.42   2.14   2.39   2.90
                           NSW   0.88    1.29   1.71   1.55                            NSW   0.77   1.13   1.44   1.44
                           NT    1.63    1.59   2.04   2.17                            NT    1.47   2.02   1.96   1.97
                           QLD   1.82    1.65   2.18   1.59                            QLD   1.54   1.64   1.69   1.57
                           SA    3.11    1.64   2.00   2.28                            SA    2.61   1.65   1.79   1.91
                           TAS   0.96    1.33   1.28   1.57                            TAS   0.94   1.33   1.28   1.27
                           VIC   2.67    2.92   2.87   2.78                            VIC   2.43   2.57   2.62   2.60
                           WA    0.96    1.25   1.47   1.34                            WA    0.79   1.32   1.25   1.25
                           AUS   1.70    1.81   2.08   1.90                            AUS   1.49   1.67   1.80   1.79
                           3.5                                                         3.5

                           3.0                                                         3.0
  Percent Exempt Grade 7




                                                              Percent Exempt Grade 9




                           2.5                                                         2.5

                           2.0                                                         2.0

                           1.5                                                         1.5

                           1.0                                                         1.0

                           0.5                                                         0.5
                                 2008    2009   2010   2011                                  2008   2009   2010   2011
                           ACT   0.92    1.45   1.48   1.39                            ACT   0.29   2.01   1.63   1.30
                           NSW   0.61    0.69   1.33   1.23                            NSW   0.53   0.71   1.33   1.18
                           NT    1.28    1.21   2.80   2.26                            NT    1.91   1.96   2.16   2.21
                           QLD   1.55    1.45   1.55   1.59                            QLD   1.29   1.52   1.53   1.36
                           SA    2.03    1.35   1.53   1.80                            SA    2.22   1.38   1.32   1.77
                           TAS   0.71    0.99   1.09   1.58                            TAS   0.62   0.95   1.41   1.22
                           VIC   1.68    1.64   1.84   1.92                            VIC   1.80   1.72   1.94   1.96
                           WA    0.96    1.25   1.42   1.19                            WA    0.64   1.51   1.28   1.61
                           AUS   1.22    1.21   1.54   1.53                            AUS   1.14   1.30   1.53   1.50




                                                                                                                         17
Appendix 3:                                Percentage of students withdrawn, Numeracy by Year,
                                           Year level and Jurisdiction
                              3.5                                                                3.5
                              3.0                                                                3.0
  Percent Withdrawn Grade 3




                                                                 Percent Withdrawn Grade 5
                              2.5                                                                2.5
                              2.0                                                                2.0
                              1.5                                                                1.5
                              1.0                                                                1.0
                              0.5                                                                0.5
                              0.0                                                                0.0
                                    2008    2009   2010   2011                                         2008   2009   2010   2011
                              ACT   0.80    2.11   3.32   4.45                                   ACT   0.60   0.70   2.37   2.24
                              NSW   0.77    0.56   0.99   0.86                                   NSW   0.56   0.35   0.61   0.55
                              NT    0.83    0.63   1.22   1.46                                   NT    0.06   0.13   0.80   0.81
                              QLD   0.33    0.68   1.55   1.51                                   QLD   0.24   0.44   1.14   1.35
                              SA    0.62    1.97   2.91   3.10                                   SA    0.43   1.43   2.16   1.99
                              TAS   0.17    0.39   1.45   1.15                                   TAS   0.11   0.31   1.17   1.23
                              VIC   0.08    1.55   2.59   2.01                                   VIC   0.07   1.13   1.85   1.51
                              WA    0.34    0.68   0.94   1.01                                   WA    0.12   0.31   0.59   0.73
                              AUS   0.45    0.95   1.68   1.52                                   AUS   0.31   0.63   1.17   1.11
                              3.5                                                                3.5
                              3.0                                                                3.0
                                                                     Percent Withdrawn Grade 9
  PercentWithdrawnGrade 7




                              2.5                                                                2.5
                              2.0                                                                2.0
                              1.5                                                                1.5
                              1.0                                                                1.0
                              0.5                                                                0.5
                              0.0                                                                0.0
                                    2008    2009   2010   2011                                         2008   2009   2010   2011
                              ACT   0.25    0.48   0.98   1.51                                   ACT   0.29   0.40   1.28   1.69
                              NSW   0.30    0.24   0.30   0.30                                   NSW   0.35   0.22   0.38   0.30
                              NT            0.03   0.38   0.61                                   NT           0.04   0.21   0.62
                              QLD   0.18    0.53   0.98   1.03                                   QLD   0.46   0.66   1.46   1.95
                              SA    0.44    1.01   2.08   1.51                                   SA    0.20   1.20   1.92   1.31
                              TAS   0.15    0.24   0.93   0.95                                   TAS   0.18   0.09   1.05   1.16
                              VIC   0.05    0.57   0.91   0.73                                   VIC   0.08   0.67   1.32   0.95
                              WA    0.12    0.51   0.35   0.55                                   WA    0.04   0.15   0.39   0.38
                              AUS   0.20    0.46   0.75   0.71                                   AUS   0.26   0.49   0.98   0.96




                                                                                                                                   18

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:0
posted:4/2/2013
language:English
pages:20