Progress and achievement in the PANDA report

Core module 3 Progress and achievement in the PANDA report Introduction This module should be tackled after Module 1 Standards in the PANDA report, which uses the same two case-study schools. You may find it helpful to look back at your work or the commentaries for that module, or to consult the table of standards from it alongside the tables for this module. This module has two parts that you should do in order: Part 1: achievement at Key Stage 2 Part 2: achievement at Key Stage 4 A commentary is provided after each task to enable you to draw comparisons with your own response or to prompt you if you get stuck. Learning objectives for this module   To be aware of the information about progress shown in the PANDA report To recognise how the PANDA report grades can help you to reach judgements on achievement To be able to interpret prior attainment (PA) comparison grades by considering the proximity of results to grade boundaries To know when and how to use benchmark tables from the Autumn Package To weigh up differences between the various progress grades To pick out the most important information about progress To identify issues for inspection from data on progress      Information sheets for this module  1: Interpreting KS1-3 average points scores Core module 3: progress and achievement in the PANDA report 1 of 32   2: Explanation of KS4 scoring system 4: Contextual data for Schools A and B Other files you may wish to refer to   Tables and graphs for core module 1 Tables and graphs for core module 3 Information about progress in the PANDA report In this module, ‘progress’ is used to refer to the measures in the PANDA that show the rises or falls in standards for the same cohort of pupils between key stages. The PANDA report contains grades for two types of information about the progress pupils make between key stages. They are based on:   prior attainment (PA) bands the value-added measure (VA). Each of them is worked out in a different way, so the grades for them may differ. The method for finding the PA grade compares one figure, the average points score (APS) for the whole cohort, with the APS for schools with roughly similar standards at the previous key stage. In contrast, the calculation for the VA measure is based on pupil-level data and compares each pupil’s progress with the progress that similar pupils made. These two measures of progress contribute to:   judgements on achievement the grade for standards compared with similar schools in the summary report. Guidance on the use of the grade for comparison with similar schools in the standards table in the summary report differs between Key Stages 2 and 4. For Key Stage 2, the PA grade should be used unless inspection evidence suggests otherwise. A change by more than one grade must be discussed with Ofsted. For Key Stage 4, where there are now grades produced by two different scoring systems, inspectors must use their professional judgement which grade to use, based on the PA grade for capped APS as a starting point. Core module 3: progress and achievement in the PANDA report 2 of 32 Part 1: achievement at Key Stage 2 Why do the grades based on prior attainment need careful interpretation? Inspectors need to interpret benchmark grades based on prior attainment with great care. The reasons for this may be illustrated by School A’s results. For Key Stage 2 pupils in School A, the statement written above table 6.4 (overleaf) shows that the average points score (APS) at Key Stage 1 was 13.9. This is about one point below the national average (for details of which see the information sheet Interpreting KS1-3 average points scores). It places the school at the top end of the 12 to 14 prior attainment (PA) band, which is the band just below the middle band. The KS1 APS can be calculated only for those KS2 pupils who have a result for any KS1 assessment recorded on the national database, whether they were assessed in School A or elsewhere. It will not include pupils who were absent or disapplied, or who were educated abroad or in the independent sector at KS1. Therefore it might not represent the actual prior attainment of all of the pupils who took the KS2 tests. A rough guide to the percentage of KS2 pupils included in the KS1 APS calculation is the percentage coverage shown above the value-added tables, which is discussed later in this module. In order to ascribe grades for Key Stage 2 standards, based on Key Stage 1 prior attainment, the Key Stage 2 results of all schools in the band are put into rank order, and the top 5% of schools are given grade A*, the next 20% grade A, and so on. If the KS2 results falls near the middle of the range for a grade, the grade is a more reliable indicator of progress than if they fall near a boundary, where they are near to another grade. Grades A* and E* tend to be more secure; even though they include only a small percentage of schools, they have a wide spread of results so most of them do not fall near the boundary with A or E and are definitely a long way from average. Core module 3: progress and achievement in the PANDA report 3 of 32 School A Benchmarks based on prior attainment at the end of Key Stage 1 The Key Stage 2 pupils in your school had a Key Stage 1 average points score of 13.9. This puts the school into the following benchmark group. Schools that achieved a Key Stage 1 average points score of at least 12 but less than 14 in 2000 Threshold table APS table In table 6.6, you can see that School A has an overall Key Stage 2 APS of 26.9, so its pupils have increased their APS from 13.9 at KS1 to 26.9 at KS2, a gain of 13 points. This is equivalent to just over two levels and has put them into the band for a grade A. Most other schools in the same 12-14 points prior attainment band which also have an increase of 13 points will fall into a lower grade band, with the lowest being graded D, for a KS2 APS of 25 (12.0 + 13). School A is therefore advantaged by being at the top of its prior attainment band, so its high grade needs to be interpreted with caution. It is useful to check what School A’s grade would have been had its KS1 APS been just 0.1 higher, at 14.0, so that it fell at the very bottom of the next KSI Core module 3: progress and achievement in the PANDA report 4 of 32 APS band and was compared with schools that had higher prior attainment than it. Task 1 shows how this is done. Task 1 overall progress Table 2.3.4 below is adapted from the Autumn Package. It is for the KS1 prior attainment band for APS 14-16, the next higher band above School A’s band. Place School A’s overall KS2 APS of 26.9 into table 2.3.4. Decide which grade or grades you now think might reflect accurately School A’s overall progress. Consider   the grade for the KS1 APS 12-14 band shown in table 6.6 above that the KS1 APS is 13.9. There is no need to consider  separate subjects. Table 2.3.4: Schools with 2000 Key Stage 1 average points score of: at least 14 but less than 16 Average KS2 points score achieved A* English Mathematics Science Overall 95% 29.3 29.4 30.9 29.6 A UQ 27.9 28.0 29.6 28.4 B 60% 27.2 27.3 29.0 27.8 C Median 26.9 27.0 28.5 27.5 C 40% 26.4 26.6 28.2 27.1 D LQ 25.7 25.9 27.6 26.5 E 5% 24.1 24.2 25.8 25.0 E* LQ is lower quartile; at 25% UQ is upper quartile; at 75% Commentary on task 1 Using table 2.3.4, the school’s overall KS2 APS of 26.9 would fall into the grade D band, slightly nearer to C than to E. Note that the Autumn Package table has more columns than the tables in the PANDA report. This is because the grade C band is split into two parts, above and below the median (middle) value. It may also be useful to consult the threshold tables for prior attainment band 14-16 in the Autumn Package to Core module 3: progress and achievement in the PANDA report 5 of 32 inform judgements on progress in subjects, particularly where there is any potential underachievement. School A’s progress may be overestimated at A, based on table 6.6, as it was advantaged by being right at the top of the 12-14 PA band. The lowest it could possibly be is D, if it were compared with the previously higher attaining schools in the 14-16 PA band. The grade likely to reflect its progress most accurately is therefore somewhere between A and D, so B or C. Subjects The PA tables include grades for each core subject. They do not show progress within the same subject, such as between mathematics at KS1 and at KS2. Instead they show progress from overall APS at KS1, so some subject detail is lost. This may be illustrated by the example of Schools X and Y. The table below shows two schools that have ten pupils with the same KS1 APS. All ten pupils in each school have the same profile of KS1 results, as shown in the table. Although pupils in both schools have the same KS1 APS, they have different levels in reading and in mathematics. The table also shows that pupils in both schools have the same KS2 results. Before reading the commentary below the table you may wish to consider whether pupils in one school have made more progress in mathematics than pupils in the other. School Reading Level X Y 3 2 Key Stage 1 results Writing Level 2 2 Maths Level 2 3 17 17 APS Key Stage 2 results Maths % of pupils at Level 4+ 90 90 Maths % of pupils at Level 5+ 30 30 Using the extract below from the Autumn Package for mathematics, each school would be graded B for progress to Level 4+ and E for progress to Level 5+. However, the pupils in School X have actually made more progress in mathematics than those at School Y. The low grade at Level 5+ for School X is more likely to reflect KS1 standards in mathematics than quality of teaching and other provision, but in School Y it may reflect weaknesses in provision as only three of the ten pupils have added the expected two levels between key stages. Core module 3: progress and achievement in the PANDA report 6 of 32 Table 2.3.5: Schools with 2000 Key Stage 1 average points score of: at least 16 but less than 18 Percentage of pupils achieving Level 4 and above A* Mathematics 95% 100 A UQ 93 B 60% 89 C Median 86 C 40% 84 D LQ 80 E 5% 67 E* Percentage of pupils achieving Level 5 and above A* Mathematics 95% 67 A UQ 52 B 60% 45 C Median 41 C 40% 38 D LQ 33 E 5% 20 E* LQ is lower quartile; at 25% UQ is upper quartile; at 75% The use of APS for all core subjects combined in the PA bands masks any subject differences in prior attainment. The PA grades for subjects must therefore be interpreted carefully. These subject differences also affect overall attainment; pupils with higher prior attainment in English tend to make more progress overall than others with similar prior attainment APS and go on to achieve a higher APS at the next key stage. This must be borne in mind when interpreting PA grades for schools where English prior attainment levels differ substantially from the levels in other subjects, such as those with a relatively high proportion of pupils who are not fluent in English. Task 2 subjects In School A, standards at KS1 for the 2004 KS2 cohort were slightly higher in reading and writing than they were in mathematics. Look back at table 6.6 for APS in School A. How would you judge the progress from KS1 to KS2 in each subject in School A? Consider  the possible overestimation of PA grades due to the high position of the KS1 APS of 13.9 in its band the PA grades for subjects if the school were in the next higher KS1 APS band, using the Autumn Package table 2.3.4 the relative standards in subjects at KS1 for this cohort, as stated at the beginning of this task.   There is no need to consider  the threshold table Core module 3: progress and achievement in the PANDA report 7 of 32  the overall APS for all core subjects. Commentary on Task 2 As School A’s KS1 APS of 13.9 puts it right at the top of its prior attainment band, the subject grades based on prior attainment (PA) may be overestimates. The APS table (6.6) suggests that pupils make much better progress in English and mathematics, with grade A, than in science, with a high grade C. However, as the KS1 attainment in the English subjects was slightly above that in mathematics, the high PA grades in English may reflect prior attainment more strongly than school provision. The high PA grade in mathematics may represent better progress than in English and reflect relative strengths in provision. Checking the subject APS using the Autumn Package table for the next higher band, APS 14-16, gives grades C for English, C/D for mathematics and E for science. They are underestimates for the progress grades because they compare School A with schools of higher prior attainment, so an accurate grade lies between them and the grades using PA band 12-14. This would suggest subject grades of B for English, a low B for mathematics and D for science. Task 3 attainment groups Look at table 6.4 for thresholds in School A. What does it show about the progress of different attainment groups in each subject? What issues for inspection does this raise? Consider   the differences in grade for different thresholds the reasons for different grades for APS in table 6.6 than for thresholds in the same subject how many more pupils in a cohort of roughly 25 would need to reach a level in order for the grade to improve (each pupil represents roughly 4% of the cohort).  Core module 3: progress and achievement in the PANDA report 8 of 32 For a reminder on interpretations about attainment groups, you may wish to look back to the section on thresholds and different attainment groups and task 4 in Module 1. Commentary on Task 3 Strengths in progress are in mathematics and for lower attainers in the school who reach the Level 4+ threshold. There may be some underachievement at Level 5+ in science and English and overall in science. It is important to remember that all PA grades may be overestimates due to the school’s high position in its KS1 APS band. The threshold tables show that the similar APS grades for English and mathematics are arrived at differently. This suggests particularly strong progress in English for lower attainers, only 11% of whom have not reached Level 4+, but below average progress for those reaching Level 5+. In contrast, the strong progress in mathematics is consistent at both thresholds, suggesting that pupils at all attainment levels make good progress in mathematics. In science, progress is below average; it is weaker for pupils attaining Level 5+ than for those reaching Level 4+. As the KS1 attainment in the English subjects was slightly above that in mathematics, the high PA grades in English may reflect prior attainment more strongly than school provision. The low grade for Level 5+ would suggest particularly weak progress. In contrast, the high PA grades in mathematics appear to represent progress rather than prior attainment in mathematics, so this subjects looks to have strong provision. Only two more pupils need to reach Level 5+ for the C grades in English and science to move up to B, and only one more pupil needs to reach Level 4+ in science for the school’s results to reach grade A. Grades should therefore be interpreted with care. Issues to follow up: ▪ causes of the strong progress for low attainers in all subjects ▪ causes of the strong progress across the attainment range in mathematics ▪ whether there is underachievement of higher attainers in science and English ▪ whether there is underachievement in science overall Core module 3: progress and achievement in the PANDA report 9 of 32 ▪ ▪ ▪ check attainment on entry to KS2, as the school’s KS2 cohort may not include as many pupils with KS1 Level 3+ as other schools with the same KS1 APS, so would naturally have fewer reaching high levels at KS2 In this small cohort the increase in level of just a few pupils leads to a higher grade, so check the individual circumstances of pupils who did not meet expected levels A very strong focus on meeting the Level 4 threshold could produce the school’s pattern of results, with high proportions reaching Level 4 but underachievement at Level 5+; check whether it is a cause of the results. Value added The value-added (VA) measure is a different way of evaluating progress from PA and is not available for separate subjects. For KS1 to KS2 it is calculated by comparing each individual pupil’s KS2 APS with the median APS nationally for all pupils with the same KS1 APS. The graph below shows the median line for the 2004 KS2 cohort. It shows that a pupil with a KS1 APS of 13 and a KS2 APS of 27 would have attained 2 points higher than the national median. Value-added median Key Stage 1 to Key Stage 2 For the whole school, the average of its pupils’ excess or deficit in relation to this line is calculated. This will come to a number near zero. After this, 100 is added, so all schools’ VA measures are clustered around 100. So 100 is the VA measure for a school in which all pupils made on average the progress shown by the median line. Table 6.8 below shows the national benchmarks for each grade. The top and bottom benchmarks are 102.0 and 98.0, so the middle 90% of schools have a difference of only four points between their VA measures. As six points Core module 3: progress and achievement in the PANDA report 10 of 32 represent a level, a VA measure of two points below 100 (98.0) represents on average one third of the pupils having added one level less than expected in all core subjects or all pupils having added one level less in one core subject. For more information on the links between points scores and levels, see the information sheet Interpreting KS1-3 average points scores. (For KS4 VA, 1000 is added instead of 100 to take account of the larger numbers produced by the new scoring system.) School A value-added table 6.8 The school’s VA grade in comparison with the national distribution, shown in table 6.8, compares School A with all others by putting their VA scores in order. School A’s VA measure of 100.6 puts it between the 75 th and 60 th percentiles of schools, just above the middle of grade B. As 0.6 points is one tenth of 6 points, on average pupils make one tenth of a level more progress than shown by the median line. Pupils in high-attaining schools generally make more progress than do pupils in low-attaining schools. In fact, the National Curriculum levels are based upon the median pupil making progress through one level every two years, but with higher attainers making faster progress and low attainers making slower progress. Therefore the VA grade for national comparisons is likely to give low grades for schools with more low-attaining pupils and high grades for schools with more high-attaining pupils. To give clearer VA information about a school in relation to similar schools, tables 6.9 and 6.10 are also produced at Key Stages 2 and 3. However they are not available at Key Stage 4. Above the value-added tables in the PANDA report there is a statement of the percentage coverage of pupil data between the key stages. For School A this is 93%, showing that for 93% of its pupils with KS2 results there were KS1 data in the national database. These pupils could have been assessed at KS1 in School A or elsewhere; coverage is not a measure of mobility. The omitted pupils may have been absent or disapplied at KS1, or been educated abroad or in the independent sector. Tables are not produced for schools with coverage below 50%. Core module 3: progress and achievement in the PANDA report 11 of 32 School A value-added tables The coverage of the matched pupil level data between KS1 and KS2 is 93%. Task 4 value added Using the value-added measure tables 6.8 to 6.10, and what you know about the possible PA grades for School A, how would you now judge progress from KS1 to KS2? Commentary on task 4 All three VA grades are B; a middle B nationally but a lower B for the free school meals and prior attainment comparisons. This shows that School A’s pupils added more value than did the median (50th percentile) school in each band: nationally, its fsm band and its PA band. Given the school’s advantaged position at the top of its KS1 prior attainment band, you might expect it to have a higher VA grade in table 6.10 than for national comparison. It does not. Instead, the low position in table 6.10 of the 100.6 VA measure in relation to the 100.4 boundary with grade C suggests that a fair reflection of the school’s progress is right on the B/C borderline. Core module 3: progress and achievement in the PANDA report 12 of 32 This B/C grade matches the information given by the PA grades which appear to show average to above average progress overall, but with variation across subject and attainment groups that includes some underachievement. Based on the data alone, a low grade B appears to reflect progress most accurately. Given the variation across subjects and attainment groups, inspection must determine whether progress is above average or average. Links between progress in PANDA report tables and achievement The information on progress in the PANDA report is based on ranking schools. It is norm referenced, so it naturally places 50% of the schools below the median average. Allowing for the middle band of 20% of schools, which are given grade C, this leaves 40% of schools in bands D to E*. Their progress or value added in these schools can correctly be described as below average. This does not automatically imply that it is unsatisfactory. For example, if all schools made just about the same progress, it would be satisfactory for each school and yet below average for half of them, but by a very small amount that had no educational significance. The judgement on achievement is a professional judgement based on the criteria in the Framework for inspecting schools . It is criterion referenced. Last year, pupils’ achievement was judged to be unsatisfactory or below in 5% of primary schools and 9% of secondary schools. The distribution of achievement grades is not expected to match that of the PA or VA grades in the PANDA report in any simplistic way. The tables below show that there is not a direct match between progress grades in the PANDA report and achievement judgements. Distribution of PANDA report grades and achievement judgements PANDA report grade % of schools achievement judgement approximate % of primary or secondary schools in 2003/4 A* 5 1&2 14 A 20 B 15 3 55 C 20 D 15 4 24 E 20 E* 5 5, 6 & 7 7 Core module 3: progress and achievement in the PANDA report 13 of 32 In the PANDA report grading system, the lowest 5% of schools are given grade E*. It is likely that the schools graded E* for PA or VA will also have the lowest achievement but care must be taken in interpreting these grades. Schools with much higher PA or VA grades may also have unsatisfactory achievement arising from the underachievement of groups of pupils, which may not show up in the PANDA report because it is compensated for by higher achievement by other groups. As only roughly 7% of schools were judged to have unsatisfactory achievement, a school with grade D for PA and VA was unlikely to have unsatisfactory achievement as were many of the schools with PA and VA graded E. You must take care not to form too early a view of a school’s achievement from looking at its PA and VA grades. Task 5 achievement Think back to all the information on progress in PA tables 6.4 and 6.6 and VA tables 6.8 to 6.10. What is your hypothesis about achievement from KS1 to KS2 in School A? Consider   whether there is underachievement in any subject or group of pupils how progress grades in the PANDA may link to inspection judgements on achievement. Commentary on task 5 There is apparent underachievement in science and at Level 5+ in English. Inspection must determine the degree of any underachievement in subjects, attainment groups and other groups. If substantial, the overall KS2 judgement would be that achievement was at best unsatisfactory, if not, that it was satisfactory. If there were found to be no underachievement in subjects or groups, achievement overall would be good. Information in the PANDA report suggests that progress is on the B/C borderline for the 2004 cohort. However, there is no direct link between PA and VA grades and achievement; achievement is a professional judgement taking groups and subjects into account, informed by the grades in the PANDA report. Core module 3: progress and achievement in the PANDA report 14 of 32 Task 6 standards and achievement in School A Look at the information in tables 3.1, 5.3 and 6.3 on standards at Key Stages 1 and 2 for School A. How does this affect your judgement in task 4 on progress and your hypothesis in task 5 about achievement? School A Key Stage 1 2004 Key Stage 2 2004 Core module 3: progress and achievement in the PANDA report 15 of 32 Commentary on task 6 It is important to check that the judgement on progress is consistent with the standards at each key stage. At first glance it appears that there might be a contradiction between progress from KS1 to KS2 being graded B or C and the 2004 KS2 standards being graded below those at KS1. In fact, table 3.1 reveals that standards at KS1 have risen substantially since 2001. They may also have been low at KS1 in 2000 for the 2004 Key Stage 2 cohort. If so, it would be consistent with satisfactory or better KS1-2 progress. However, different pupils from those assessed at KS1 may have taken the KS2 tests. Progress for the 2004 KS2 cohort and for current pupils will need to be checked during inspection. The summary report must make clear why the standards and progress judgements are consistent so that readers do not perceive any contradiction. The progress grades (PA and VA) at KS2 are above the standards grades at KS2. Standards at KS2 are roughly average in English and mathematics, in which the PA grades for APS, shown in table 6.6, are A. In contrast, standards in science are well below average, and the PA grade is C. These grades and the disparity between subjects suggest that there is underachievement in science, leaving pupils with low results. The relative strengths and weakness of subjects and at thresholds will naturally be consistent in the tables for standards and progress. However, the PA table 6.4 gives an indication of whether the relatively low standards in all subjects at the higher thresholds represent above or below average progress. It shows that, for mathematics, the progress is strong with the top quarter of pupils reaching Level 5+. It also indicates potential underachievement in science and English for two reasons: the likely overestimation in the PA grades, and the higher attainment in English subjects for this cohort at KS1. It must be noted that, in this small cohort, grades would rise if just a few pupils obtained one level higher, and individual circumstances may have affected some pupils’ test performance. Inspectors should take this into account when judging achievement. Core module 3: progress and achievement in the PANDA report 16 of 32 Key questions on progress to ask yourself when interpreting the PANDA report data The main considerations a What is the educational significance in terms of levels/grades and proportions of pupils? b How near to boundaries are the grades? Checking the validity of grades c Does the school’s previous APS put it above or below the national average? d Does the percentage coverage show a high proportion of pupils for whom no data on progress are available? e Does the position of the school’s APS in the previous key stage prior attainment band tend to boost or reduce its PA grades? f Is the previous key stage APS so near a boundary that I should use the Autumn Package to check the grades for the adjacent band? PA step by step g What are the PA grades overall? h What are the differences in PA grades for subjects and could these be due to different attainment at the previous key stage? i What are the differences in PA grades for thresholds and what do the percentages show about the attainment groups reaching them? Overall step by step j k l Are there differences between the three VA grades and how do the school’s PA and FSM bands contribute to them? Are there differences between VA and PA grades? How may the school’s previous key stage APS affect them? Is the overall grade for progress consistent with standards for the cohort at the beginning and end of the key stage? Are these standards any different from current standards at these key stages? Also at KS4 m Are there any differences between the grades using each scoring system at KS4, or between capped and uncapped APS? n Are there differences between the progress for this cohort at each key stage in PA grades or in KS3-4 and KS2-4 VA grades? The remainder of the module relates only to Key Stage 4. For readers who do not wish to work through this part of the module, the module review points are included here. Core module 3: progress and achievement in the PANDA report 17 of 32 Review of module Identify the extent to which the module has enabled you to meet its objectives:   To be aware of the information about progress shown in the PANDA report To recognise how the PANDA report grades can help you to reach judgements on achievement To be able to interpret prior attainment (PA) comparison grades by considering the proximity of results to grade boundaries To know when and how to use benchmark tables from the Autumn Package To weigh up differences between the various progress grades To pick out the most important information about progress To identify issues for inspection from data on progress      Core module 3: progress and achievement in the PANDA report 18 of 32 Part 2: achievement at Key Stage 4 School B The first four tasks are about case-study school B from Module 1 Standards in the PANDA report. Look back to the commentary on the final task in that module, task 11, to remind yourself of its standards and the issues to follow up in inspection. You may also find it helpful to look at the information sheet Contextual data for Schools A and B and at a printout of Tables and graphs for core module 1. Table 3.3 in the PANDA report, below, gives the only overview of changes in progress in the last four years; there are no graphs that show this. You can also use it to check the progress of the 2004 KS4 cohort when it was the 2002 KS3 cohort. School B 1 These results include a wider range of qualifications such as Key Skills, BTECH and Vocational qualifications. 2 Benchmarks for 5+A*-G from 2002 onwards only include results for pupils achieving the threshold where both English and mathematics are among their A*-G grades. Core module 3: progress and achievement in the PANDA report 19 of 32 Task 7 overall progress in the last four years Look at table 3.3 for School B. Using only table 3.3, how would you judge the progress of the 2004 KS4 cohort? What issues for inspection does this raise? Consider    its progress at KS3 changes in the progress of KS4 cohorts in the last four years causes of any relative weaknesses, bearing in mind footnote 2 in table 3.3. There is no need to consider  subjects or courses. Commentary on task 7 The first thing to consider is whether all four of the points score grades are the same: the capped and uncapped in each scoring system. For School B they are all A. Overall progress for the 2004 KS4 cohort appears to be very good, but we have not yet checked the position of the cohort’s KS3 APS within its prior attainment band to find out whether the PA grades might be overestimates or underestimates. Whilst uncapped APS has remained strong, the capped APS has improved in the last two years showing that pupils are on average progressing to higher grades than before in their best 8 subjects. Inspection should tease out and report the factors that have contributed to this success. There are relative weaknesses in reaching the 5+ A*-G threshold. Upon the introduction in 2002 of the requirement to include English and mathematics results at this threshold, the school’s progress dropped and has since been rising gradually. Inspection should investigate the causes of the improvement and why it has not been greater. Core module 3: progress and achievement in the PANDA report 20 of 32 The progress of the 2002 KS3 cohort appears to be very good, so assuming that it contained mainly the same pupils as the 2004 KS4 cohort, the progress from KS2 to KS4 appears also to be very good. Making fuller judgements of progress In order to make a fuller judgement of progress and to inform your judgement on achievement, you need to take into account the more detailed data in the PANDA report on:    standards in subjects grades based on prior attainment grades based on value added. Checking prior attainment The KS3 APS for the KS4 cohort is the first piece of information to check so you can evaluate the attainment at entry to the key stage. This helps you to interpret the KS3-4 VA grade, which compares the school with all others nationally and may consequently be lower than the PA grade for a school cohort with low prior attainment. It also helps you to judge where within the KS3 PA band the school’s APS lies and therefore whether the PA grades are likely to be overestimates or underestimates of progress. For School B the KS3 average points score was 32.1 for the 2004 KS4 cohort. This puts its attainment on entry to KS4 about 1.7 points, or roughly a quarter of a NC level per pupil on average, below the national average of 33.8. You can look up the national results and prior attainment bands in the information sheet Interpreting KS1-3 average points scores. The KS3 APS of 32.1 puts the school in the top third of the 30-33 prior attainment band. This advantages it very slightly, giving it marginally higher grades than might reflect its actual progress. The KS2-4 VA measure takes account of long-term progress, during which time there may have been a number of major changes in the school. You should bear in mind that it may be influenced strongly by provision in the school four or five years ago. Core module 3: progress and achievement in the PANDA report 21 of 32 Task 8 grades based on value added Look at tables 6.6 and 6.7 for value added in School B. How would you now judge the progress of the 2004 KS4 cohort? Consider   percentage coverage how close to grade boundaries the school’s VA measure falls and what this means in terms of GCSE grades for a cohort of roughly 150 pupils reasons for the lower KS3-4 VA grade than the PA grade of A, bearing in mind the cohort’s KS3 prior attainment.  School B Comparison with National Benchmarks: Value-added Measure between KS2 and KS4 The coverage of the matched pupil level data between KS2 and KS4 is 91%. Comparison with National Benchmarks: Value-added Measure between KS3 and KS4 The coverage of the matched pupil level data between KS3 and KS4 is 94%. Core module 3: progress and achievement in the PANDA report 22 of 32 Commentary on task 8 The first thing to consider is whether both VA grades are the same and whether they match the PA grades, all of which are A. The grade for KS2-4 is a high A and for KS3-4 a middle B. The overall progress still appears to be very good, but the lower grade for KS3-4 VA than for PA suggests that closer evaluation of achievement for pupils at the extremes of attainment should be followed up. The high percentage coverage for both VA measures shows that the grades give a fairly full picture of the progress of the whole cohort, but inspection will need to check that pupils omitted from the VA and PA calculations are also making equivalent progress. For KS3-4 there are 6% omitted from the VA calculation, or 9 pupils in a cohort of roughly 150. At KS3 they may have been absent, disapplied or educated abroad or in the independent sector. For KS2-4 VA, 9% of pupils are omitted. For KS3-4 the VA grade is near the middle of the band for grade B. This is below the PA grades which are A. There are three potential causes of this: ▪ VA compares School B with all schools nationally, rather than only those in its PA band, which is just below national average ▪ PA grades may be slightly overestimated ▪ differences in calculation methods: added value for very high or low attainers in a school that is unrepresentative of its value added for other pupils may cause differences between VA and PA grades and suggest that achievement of these pupils is an issue to follow up. The KS2-4 VA grade is a particularly high A showing close to excellent value added. Given what we know about the school’s below average attainment for this cohort at KS3 and this implication for its attainment at KS2, this high grade in comparison with all schools nationally gives even a stronger impression of the value added from KS2 to KS4. The cause of this very strong KS2-4 value added should be teased out and reported, and the reasons for the lower valueadded grade from KS3-4 explored. The VA grades are based on the capped total points score and take into account all GCSE and equivalent qualifications. The information sheet Explanation of KS4 scoring system shows the equivalence between full GCSE grades and differences in capped total points score. The KS3-4 VA measure is roughly 4 points below the boundary with the band for grade A. This gap could be closed by roughly two thirds of pupils gaining one grade higher in one subject. The KS2-4 VA measure is roughly 9 points away from the A* band so about twice as much improvement as for the KS3-4 measure would be needed to increase the grade, for example two thirds of pupils gaining one grade higher in two subjects. Core module 3: progress and achievement in the PANDA report 23 of 32 Task 9 grades based on prior attainment Look at tables 6.4a and 6.4b for standards based on prior attainment. What further information does this give you about the progress of the 2004 KS4 cohort in School B? Consider   reasons for some PA grades being higher than others how close to grade boundaries the school’s results fall and what this means in terms of GCSE grades for a cohort of roughly 150 pupils. Benchmarks based on prior attainment at the end of Key Stage 3 Table 6.4a uses the old scoring system. Table 6.4b uses the new scoring system. Your school had a Key Stage 3 average points score of 32.1. This puts you into the following benchmark group. Core module 3: progress and achievement in the PANDA report 24 of 32 Commentary on task 9 Progress at KS4 overall is very good, with strengths at the 5+ A*-C threshold and greater strengths in uncapped than capped APS. However there are still relative weaknesses for lower attainers in reaching the 5+ A*-G threshold even though there have been big improvements since 2002. It is important to check whether the PA grades may over or underestimate progress. In this case they may be a slight overestimate. The Key Stage 3 APS of 32.1 is below the national average but it puts the school in the top third of the 30-33 prior attainment band. This advantages it slightly, giving it higher grades than might reflect its actual progress. However, none of the four KS4 APS falls near the bottom of its band, so all are strong grade As. The capped APS is near the middle of the grade A band, whilst the 57% of pupils gaining 5+grades A*-C are right at the top of the grade A band. This lower position for the capped APS than for the threshold is contributed to by the relatively low proportion of grades A* and A. As there are roughly 150 pupils in the cohort, each one represents roughly 0.7% of the cohort. Only one more pupil would need to reach the 5+ A*-C threshold for the school’s grade to improve to A*, and only a maximum of two pupils would need to meet the 5+ A*-G threshold for the grade to improve to B. The uncapped APS is higher in its band than the uncapped APS, with a very high A in both scoring systems, due to the contribution of GNVQ. If there were differences between the scoring systems, these could arise because the new system gives greater recognition for quantity than previously. For example, if you improve in one full GCSE subject by one grade you gain an extra 6 points, but if you take one extra GCSE and obtain a grade G you gain 16 points. Under the old system, each would provide an extra one point. These differences are shown in the information sheet Explanation of KS4 scoring system . Task 10 achievement and issues Using all of the information about the 2004 KS4 cohort, what is your hypothesis about achievement at KS4 in School B? What issues for inspection does this raise? Consider   your work in Module 1 on the standards in School B the summary of standards and issues to follow up in inspection in the commentary for task 11 of Module 1. 25 of 32 Core module 3: progress and achievement in the PANDA report Commentary on task 10 Progress at KS4 overall is very good. There are relative weaknesses for lower attainers, particularly in reaching the 5+ A*-G threshold. One cause of this is the low percentage of pupils reachi ng grade G in mathematics, as explained in task 8 in Module 1. There are also relative weaknesses in attainment in science, French and German, and amongst highest attainers. Inspection should determine whether these, or any other lower achievement in subjects or by groups of pupils, would bring the achievement judgement down to good or below. A secure hypothesis from the data is that achievement is good. Issues: ▪ reasons for success at the 5+ A*-C threshold, given that there does not appear to be a focus at GCSE on reaching grade C ▪ reasons for the success in uncapped APS and improvement in capped APS ▪ reasons for relative strengths in English results ▪ whether there is underachievement at 5+ A*-G and how lower achievement at this threshold is being addressed ▪ whether there is underachievement in sciences, French and German ▪ how well the curriculum provision in sciences, French and German meets pupils’ needs ▪ whether there is underachievement by the lowest attainers in mathematics ▪ whether there is underachievement at grades A*-A. Inspection will need to check: ▪ the curriculum provision that enables standards at 5+A*-C to be above average while standards at A*-C are significantly below average in most GCSE subjects ▪ the contribution of GNVQ to APS and meeting of the Level 2 threshold ▪ the contribution of mathematics to meeting the 5+A*-G threshold. Core module 3: progress and achievement in the PANDA report 26 of 32 Judging achievement in another school The following task offers you the opportunity to apply all of the skills covered in this module to a school with a completely different context. Task 11 standards and achievement in School Z Look at all of the information on the following pages about School Z. Look also at the list of key questions on progress to ask yourself when interpreting PANDA report data, which is at the end of Part 1 of this Module. It will take you through the task step by step. How would you evaluate KS4 standards and KS3-4 progress? What is your hypothesis about achievement at KS4? Consider    the causes of the differences in grade between the two scoring systems the quantity of courses pupils take the grades for national and PA comparison for grades A*-C in the core subjects, using the Autumn Package benchmark tables (bear in mind that they are for percentages of the cohort). 1 These results include a wider range of qualifications such as Key Skills, BTECH and Vocational qualifications. 2 Benchmarks for 5+A*-G from 2002 onwards only include results for pupils achieving the threshold where both English and mathematics are among their A*-G grades. Core module 3: progress and achievement in the PANDA report 27 of 32 One quarter of the pupils take a BTEC Level 2 course. This is equivalent to two full GCSE courses. As a Level 2 course, it contributes to both the Level 2 (5+ A*-C) and Level 1 (5+ A*-G) thresholds. It also contributes to APS. The percentage of pupils taking at least one GCSE full course is 98%. Number of GCSE full courses taken by percentage of pupils GCSE subject English Mathematics Science double % A*-C grades 28.2 11.7 10.15 % A*-G grades 69.9 86.4 93.2 entry as % of cohort 91.3 96.3 89.4 % A*-C grades in cohort 25.7 11.3 9.1 % A*-G grades in cohort 63.8 83.2 83.3 2004 Autumn Package benchmark tables for percentage of pupils in cohort All schools Secondary schools with 2002 Key Stage 3 average points score of: at least 27 but less than 30 LQ is lower quartile; at 25% UQ is upper quartile; at 75% Core module 3: progress and achievement in the PANDA report 28 of 32 Benchmarks based on prior attainment at the end of Key Stage 3 Table 6.4a uses the old scoring system. Table 6.4b uses the new scoring system. Your school had a Key Stage 3 average points score of 28.8. This puts you into the following benchmark group. Comparison with National Benchmarks: Value-added Measure between KS2 and KS4 The coverage of the matched pupil level data between KS2 and KS4 is 96%. Comparison with National Benchmarks: Value-added Measure between KS3 and KS4 The coverage of the matched pupil level data between KS3 and KS4 is 94%. Core module 3: progress and achievement in the PANDA report 29 of 32 Commentary on task 11 Standards Standards are very low on all APS measures except the uncapped APS for GCSE and equivalent qualifications where they are well below average. This grade takes account of the volume of the BTEC course, which is equivalent to two GCSEs. The inclusion of the BTEC course has also raised standards at the Level 2 threshold from 13% of pupils attaining grades 5+ A*-C when only GCSE/GNVQ are counted to 22%, but does not improve the grade which remains E*. Standards have fallen this year in relation to national standards. Using the Autumn Package, GCSE standards in the core subjects at grades A*-C are very low in mathematics and science, and well below average in English. The percentage of the cohort reaching grade G in English is particularly low at just under two thirds; it prevents over one third of the pupils from reaching the 5+ A*-G threshold. However, there is a high entry for double science, and 83% of the cohort attains at least grade G, providing them with two fifths of the requirements of the Level 1 threshold. Almost 50% of pupils take fewer than 8 GCSE courses which, for those that do not take the BTEC course, will not enable them to contribute points from 8 subjects to the capped APS. Ten percent of pupils take fewer than 5 GCSE courses, so if they do not take the BTEC course they cannot meet the Level 1 threshold (5+ A*-G) or the Level 2 threshold (5+ A*-C). Multiplying the capped APS by 6 and subtracting it from the new capped APS gives roughly 73. This shows that pupils on average contributed the results of 7.3 full GCSE equivalent courses to their capped total score. Possibly 30% contributed eight while 70% contributed 7. This method is described in the information sheet Explanation of KS4 scoring system. Progress There is inconsistency between the PA grades for the two scoring systems. The grade using the old system matches the VA grades. Progress is below average or well below average from KS3 to KS4, and well below average from KS2 to KS4. Using the old scoring system, the PA tables give grade E for progress. This may be a very slight overestimate as the KS3 APS is about one third from the top of the 27-30 prior attainment band. However, both of the KS4 APS are Core module 3: progress and achievement in the PANDA report 30 of 32 near the top of the grade E band so the PA grade is a relatively secure E. This is consistent with the VA grades, both of which are in the top half of the E band. When considering grades A*-C in the core GCSE subjects, you must first work out the percentage of the cohort that reaches the threshold; the PANDA report gives only the percentage of candidates entered who meet the threshold and the percentage of the cohort that was entered. Usually you must multiply these together to find the percentage of the cohort that reached the threshold, but the result was provided for you in this task. The PA grades are a high D for English, low E for mathematics and middle E for science. Overall, pupils’ progress in the core subjects is well below average with comparative strengths in English at these higher grades, but English results are comparatively low at grade G. The BTEC course has the effect of raising the PA grade for uncapped APS to a secure C and the capped to a high D. However, progress at the 5+ A*-C threshold is D so a representative grade for progress using the new scoring system would be D. This then leaves an inconsistency between the PA grade under the new scoring system (D) and the VA grade together with the PA grade using the old scoring system (E). It is a difference generated by the two scoring systems, in which the new one ascribes more weight to quantity of qualifications than did the old one. In determining the grade for use in the similar school comparison in the standards table of the summary, inspectors will need to ensure that it gives the fairest reflection of the school. Achievement In evaluating achievement, a key element must be achievement in the core subjects, which in turn must be mindful of standards in these subjects. The data support an hypothesis that achievement is no better than unsatisfactory. Core module 3: progress and achievement in the PANDA report 31 of 32 Review of module Identify the extent to which the module has enabled you to meet its objectives:   To be aware of the information about progress shown in the PANDA report To recognise how the PANDA report grades can help you to reach judgements on achievement To be able to interpret prior attainment (PA) comparison grades by considering the proximity of results to grade boundaries To know when and how to use benchmark tables from the Autumn Package To weigh up differences between the various progress grades To pick out the most important information about progress To identify issues for inspection from data on progress      End of module Core module 3: progress and achievement in the PANDA report 32 of 32