Guide to Post-16 Contextual Value Added (CVA) 2007
Achievement and Attainment Tables
January 2008
Achievement and Attainment Tables: Guide to post-16 Contextual Value Added (CVA)
Contents
1. Introduction 2. Factors used in the CVA Model
2.1 Student Prior Attainment 2.2 Student Volume of level 3 Qualifications 2.3 Student Qualification Route 2.4 Interaction Variables 2.5 Student Gender 2.6 Cohort Prior Attainment
Page 3 Page 4
3. Student CVA Calculation
3.1 Transforming the QCA Point Score 3.2 Actual and Predicted Attainments 3.3 Capping Predictions 3.4 Using the Ready Reckoner
Page 7
4. Institutional CVA
4.1 Shrinkage Factor 4.2 Confidence Intervals
Page 9
2
Achievement and Attainment Tables: Guide to post-16 Contextual Value Added (CVA)
1. Introduction
1. The main purpose of the post-16 contextual value added (CVA) model being developed for the achievement and attainment tables is to measure the relative contribution that an institution has made to students’ performance between Key Stage 4 and the end of their sixth form studies. 2. The students included in the CVA model are the same as those included in the main post-16 tables, provided they have prior attainment data. Post-16 attainment is measured at the end of sixth form (- also referred to as the end of ‘Key Stage 5’). The students included in this definition are those who have progressed from Key Stage 4 to study level 3 qualifications – usually, but not exclusively, over a 2 year period - before moving to, for example, Higher Education or employment. All QCA approved level 3 qualifications are used to determine the student's attainment at Key Stage 5. 3. The CVA model was first piloted in 2006. The pilot was extended to allow further refinement in response to stakeholder feedback. The 2007 model includes fails grades and uses a transformed scoring system for qualifications. General Studies A and AS levels are excluded from the CVA calculations, but are still included in the main post-16 tables indicators. 4. The CVA model predicts each student’s level 3 total points score based on the national associations between Key Stage 5 outcomes, prior attainment at Key Stage 4 and other characteristics. Each student’s actual outcome is then compared with the national average appropriate to them: if their performance was higher than their peers, their CVA score will be positive, otherwise it will be negative. The individual student CVA scores are then averaged within each institution to provide the institution’s overall CVA measure. 5. To calculate the individual student and institutional CVA scores a statistical modelling technique called multi-level model (MLM) is used. Individual student CVA scores and institutional calculations can be checked using the 2007 post-16 Ready Reckoner made available with this guide.
3
Achievement and Attainment Tables: Guide to post-16 Contextual Value Added (CVA)
2. Factors used in the CVA Model
6. The post-16 CVA model predicts a student’s level 3 total point score based on factors shown to have a significant effect. 7. The factors included in the 2007 CVA model are: student prior attainment at Key Stage 4 including their achievement in GCSE English student volume of level 3 qualifications student qualification route student gender; and cohort prior attainment
8. For each factor, a coefficient measures the size of the factor’s effect on the level 3 prediction. The sum of all factors multiplied by their respective coefficients gives a prediction of the student’s level 3 total point score. This prediction is based on an average of all students with the same CVA characteristics. The difference between this prediction and the student’s actual achievement is the student’s CVA score. Individual students’ CVA scores are then averaged to produce an institution CVA score.
2.1 Student Prior Attainment
9. Key Stage 4 Prior Attainment is the most significant predictor of level 3 total point score. A student’s prior attainment is calculated as their uncapped Key Stage 4 Average Points Score based on all level 2 qualifications approved for use pre-16. 10. For 171 year olds, prior attainment is cumulative attainment in qualifications at level 2 up to and including age 15 2 . For 181 year olds, prior attainment is cumulative attainment in qualifications at level 2, up to and including age 162. 11. Note that some level 3 qualifications, e.g. AS levels, taken early and counted as an output in the Key Stage 4 Tables are not included as prior attainment when calculating CVA. 12. The formula used to calculate prior attainment is:
Prior Attainment Total Points Score at KS4 Total Volume of Entries at KS4
13. Other significant factors related to prior attainment included in the model
1 2
Age of student as at 31 August 2006 This age refers to the age at the start of the academic year. For example, a student of ‘Age 15’ is likely to be in the final year of compulsory education and will be aged 15 as at 31 August of the appropriate academic year.
4
Achievement and Attainment Tables: Guide to post-16 Contextual Value Added (CVA)
are: (Prior Attainment)2 and (Prior Attainment)3 - this is to achieve a better match of predictions to the Key Stage 5 outcome. Prior Attainment in GCSE English – analysis shows that a student’s grade in GCSE English has an impact on their subsequent attainment at level 3. The model includes a factor for the difference between a student’s point score for GCSE English and their average point score for all GCSEs (and equivalent). This variable is called English Difference in the CVA model and is calculated as:
(GCSE English Points Score) minus (Average Point Score at KS4)
If a student has not attempted GCSE English we set English Difference equal to zero.
2.2 Student Volume of level 3 Qualifications
14. The amount (quantity) of level 3 qualifications taken by a student is an equally important aspect of their success as the grade (quality) attained. By including the volume of level 3 qualifications taken as a factor in the model, we ensure one aspect does not dominate the other. 15. For the purposes of calculating the total volume of qualifications taken by a student, level 3 qualifications have been assigned a ‘size’ relative to an A level. In this way, an A level has a volume of one, an AS a volume of a half, an IB Diploma a volume of five, and so on.
2.3 Student Qualification Route
16. Analysis has shown that Qualifcation Route too is an important factor in predicting a student’s level 3 total point score. 17. Five Qualification Routes have been defined for the purpose of the pilot as follows (the numbers in brackets indicates the proportions of students in each route based on 2007 data): IB (0.5 %) AVCE (5.7 %) BTEC/OCR (17.8%) NVQ/VRQ (4.0 %) A/AS Level (71.9 %) 18. Every qualification taken contributes to the volume (see para 15) in one of these qualification routes. A student is defined as belonging to a specific qualification route if the combined volume of all the exams taken by that student is largest in that qualification route. 19. If a student has an equal maximum volume in more than one
5
Achievement and Attainment Tables: Guide to post-16 Contextual Value Added (CVA)
qualification route, then out of those maximum volume routes they are placed in that route which comes first in the ordering IB, AVCE, BTEC, NVQ, A level.
Examples of Defining the Qualification Route A student with 3 A Levels (volume = 3) and 1 Applied GCE Double Award (volume = 2) falls into the A Level Route (60% of the volume is A Level). A student with a BTEC National Certificate (volume = 2) and 1 A Level (volume = 1) falls into the BTEC Route (67% of the volume is BTEC). A student with a BTEC National Certificate (volume = 2) and 2 A Levels (volume = 2) falls into the BTEC Route (the volume is equal in those two routes, and BTEC comes before A levels in the list).
2.4 Interaction Variables
20. Prior Attainment, Volume and Qualification Route are not independent of each other – as one changes another changes. To take account of these dependencies the model includes interactions between the following factors: Prior Attainment and Qualification Route: students with a higher prior attainment tend to follow A Level and IB routes Prior Attainment and Volume: students with a higher prior attainment are likely to take a larger volume of qualifications Volume and Qualification Route: students taking an IB will have a volume of 5 whereas students taking A Levels are likely to have a volume of between 3 and 4
2.5 Student Gender
21. Analysis shows that females tend to achieve a higher level 3 total point score than males at the end of Key Stage 5.
2.6 Cohort Prior Attainment
22. The model includes the average Key Stage 4 Prior Attainment of the cohort as a factor. This is known as the peer effect and is calculated as the average prior attainment of all students at the end of Key Stage 5 at an institution. Students tend to perform better if they are in a group of students with high prior attainment.
6
Achievement and Attainment Tables: Guide to post-16 Contextual Value Added (CVA)
3. Student CVA Calculation
3.1 Transforming the QCA Point Score
23. In order to reduce a disproportionate effect of fail grades on the CVA calculation, we have transformed the point scores used in the 2007 model. This is for the purposes of calculating CVA scores only. ‘Raw’ scores calculated for the main post-16 tables will be based on the standard QCA point score system used for some time. Transforming the point scores for the CVA calculation does not affect interpretation of the resultant CVA score. This is because the relative values assigned to pass grades are the same as for the standard QCA point score system. 24. Transforming QCA point scores is achieved by subtracting 120 points for each A Level equivalent taken. For example: Examples of transforming the QCA point score
Student A takes 3 A Levels and gets grades C, D and U. QCA points = 210 + 180 + 0 = 390 Subtract 120 for each A level equivalent Transformed points = 390 – (2 x 120) = 150
Student B takes an IB and achieves grade 45 QCA points = 1380 Subtract 120 points for each A level equivalent Transformed points = 1380 – (5 x 120) = 780
25. It is important to note that, when using the Ready Reckoner, you should enter the standard QCA point score achieved by students. The Ready Reckoner will transform their point score for you.
3.2 Actual and Predicted Key Stage 5 Attainment
26. The CVA score of a student is: (Actual Attainment) minus (Predicted Attainment) Both the actual and predicted attainment are calculated using transformed point scores. 27. The Actual Attainment is the student’s level 3 total point score , i.e. the
7
Achievement and Attainment Tables: Guide to post-16 Contextual Value Added (CVA)
total transformed points derived from all the approved level 3 qualifications they have taken. 28. The Predicted Attainment is what the model predicts as the student’s transformed level 3 total point score with a given set of factor values. It is given by: (Constant) + (Average Prior Attainment × Coefficient) + (Average Prior Attainment Squared × Coefficient) + (Average Prior Attainment Cubed × Coefficient) + (English Difference × Coefficient) + (Student KS5 Volume × Coefficient) + (Student Qualification Route × Coefficient) + (Student Gender × Coefficient) + (Cohort Prior Attainment × Coefficient) + (Interaction between Average Prior Attainment and Qualification Route × Coefficient) + (Interaction between Average Prior Attainment Squared and Qualification Route × Coefficient) + (Interaction between Average Prior Attainment and KS5 Volume × Coefficient) + (Interaction between KS5 Volume and Qualification Route × Coefficient).
3.3 Capping Predictions
29. The model works well for the vast majority of students, but at the extreme ends the formula above can sometimes produce predictions that are too high or too low. We therefore cap predictions so they are never below zero (as that is obviously the minimum a student can achieve), and never above the maximum that a student could have attained for the volume and type of qualifications taken. 30. The maximum is calculated for each individual student based on the exact qualifications they took. For example, an A Level has a maximum of 270 points, and an IB has a maximum of 1380 points for IB grade 45 (before the QCA points score is transformed).
3.4 Using the Ready Reckoner
31. The ‘Student CVA worksheet’ on the Ready Reckoner can be used to calculate your individual student’s transformed level 3 total point score, their predicted transformed level 3 total point score, and their resultant CVA score which includes points capping. The ‘Institutional CVA’ page can be used to calculate your institution’s CVA score based on individual student scores, while ‘Your CVA Summary’ provides a breakdown of the institutional CVA score by student characteristic.
8
Achievement and Attainment Tables: Guide to post-16 Contextual Value Added (CVA)
32. The size of the coefficients varies from factor to factor (e.g., the coefficient for volume is different to the coefficient for qualification route). The full list of coefficients for the 2007 CVA pilot is shown in the Ready Reckoner on the ‘Coefficients’ worksheet. 33. The effect of the coefficients cannot be seen simply by looking at a list of coefficients. This is because the coefficient of a particular factor does not give the overall picture. For example, the coefficient of the Prior Attainment factor does not indicate the overall effect of prior attainment, as we have (Prior Attainment)2, (Prior Attainment)3 and interaction variables that all must be considered together to see the effect of prior attainment. 34. The Ready Reckoner can help you to understand how your institution’s and your students’ CVA scores have been derived. It cannot be used as a tool to predict what CVA students might achieve in the future.
9
Achievement and Attainment Tables: Guide to post-16 Contextual Value Added (CVA)
4. Institutional CVA
4.1 Shrinkage Factor
35. The shrinkage factor is an integral part of the model and it is determined by the number of students in the CVA calculations for an institution (see Fig. 1 for a range of shrinkage factors applied to the institutions in the 2007 model). The shrinkage factor leads to a better estimate of contextual value added for institutions with a small number of students.
36. The Ready Reckoner (Institutional CVA worksheet) enables the calculation of an institution’s shrinkage factor. 37. The final CVA score for an institution is calculated by multiplying the average of all student CVA scores in that institution by the shrinkage factor for that institution.
4.2 Confidence Intervals
38. The CVA measure is a statistical means of assessing how effective an institution is at Key Stage 5 by measuring students’ progress. We use 95% confidence intervals to show the range within which we can be confident the CVA measure represents the overall effectiveness of an institution at Key Stage 5. 39. Like the shrinkage factor, the size of the confidence interval is determined by the number of students in the CVA cohort, as shown in Fig. 2.
10
Achievement and Attainment Tables: Guide to post-16 Contextual Value Added (CVA)
40. The Ready Reckoner (Institutional CVA worksheet) enables the calculation of an institution’s 95% confidence interval. 41. The confidence intervals allow an interpretation of whether an institution’s CVA score is significantly above or below the national average. For example, if an institution’s CVA score is above the national average but its confidence interval crosses the national average (1000), then the interpretation is that the progress of its students is not statistically significantly different from the national average (see Fig. 3).
Figure 3: How to Read Confidence Intervals
Significantly above average Not significantly different from the average
National Average
Significantly below average
42. The CVA score must always be interpreted alongside its 95% confidence interval, otherwise it can lead to potentially misleading conclusions. 43. Following standard tables practice, 1000 is added to all CVA scores and their respective 95% confidence intervals for presentational purposes.
11