We recently launched a free benchmarking tool for English state secondary schools which distills the large amount of available data down into a few simple, intuitive visualisations. Our aim is to help schools, parents and governors to better understand their school’s performance in context.
We have featured 6 performance measures in the tool: new headline measures (Progress 8, Attainment 8, Ebacc and A*-C in English and Maths) plus Best 8 Value Added and Best 8 Average Point Score (APS) per Pupil, since they are the closest predecessors of Progress 8 and Attainment 8 for those schools that didn’t opt in.
We began to explain the logic behind these curation choices in the blogpost that accompanied the launch of the tool. This article goes into more detail about why we have prioritised progress over attainment and averages over thresholds. We have made the graphs screenshotted below available as a free public tool, so feel free to play with the data yourself.
How To Categorise Performance Measures
Before explaining our preferences, it is worth taking a moment to explain the types of measures that are available. Performance measures can be classified along two dividing lines: whether they relate to progress or attainment and whether they are an average or a threshold.
*measures included within our tool
In our view, not all performance measures are equal. We think that those in the top right represent the best measures of a school’s performance, while those in the bottom left are the least useful for that purpose. That is not to say that any one measure will ever provide a rounded picture of school performance on its own. We include measures from 3 of the 4 categories in our tool, and advise you to look at them all, but there is a deliberate priority order in the way the reports are structured.
Why We Prioritise Progress
A good way to illustrate why we prioritise prioritise progress measures over attainment measures is to compare one against the other. This is the result of plotting mainstream state secondaries’ 5+ A*-C including English & Maths (AC5EM) against their Best 8 Value Added:
Three things are worth noting.
Observation #1: There is a positive correlation between the two measures; as the trend line shows, the general pattern is that schools with a higher AC5EM have a higher Value Added. This is not particularly surprising: we’d expect that schools who score highly on an attainment threshold are more likely to have added value to their students.
Observation #2: Within each of the three Best 8 Value Added bands (below, at, and exceeding expected progress) there is a huge range of AC5EM pass rates. This means that knowing a school’s AC5EM is not going to give you a particularly good indication of their Best 8 Value Added and vice versa.
Observation #3: At the top of the graph, a group of schools have broken away from the rest. These schools have a very high AC5EM (above 90%), but aren’t getting similarly high Value Added. Their Value Added scores aren’t bad - all but one are making expected or above expected progress - but they also aren’t spectacularly high.
So why aren’t these schools getting top Value Added scores to match their perfect or near-perfect AC5EM scores, as predicted by the trendline? The answer becomes clearer if we colour the graph by prior attainment (Key Stage 2 average points score, converted into a fine level) rather than Value Added band:
This group at the top has very high average prior attainment, equating to a 4a or 5c in KS2 SATS. Unsurprisingly, these schools are also mostly selective. The issue is that for such schools, the AC5EM bar is a very low one. So the students’ high start point means that this threshold can be achieved even without adding much value. Moreover, DFE matrices dictate that students with a level 5 should get a B at GCSE in order to have made expected progress. So for these schools it’s actually possible to reach this attainment threshold even if students make less than expected progress. Equally, it’s hard to demonstrate stellar progress when you’re being judged by whether you’re getting high ability students over such a low hurdle.
The table below shows the diversity of AC5EM results for schools at each KS2 level and each Value Added band. As an example, if you look at those schools with an average KS2 level of a 4c (the largest prior attainment group), you will see that schools with above expected progress have AC5EM results ranging from 44% to 95%.
The lowest, highest and average AC5EM passrate of schools within each prior attainment and value added band
It is also helpful to look at the percentage of students who actually go on to attain the AC5EM threshold for each Key Stage 2 (KS2) baseline level:
|Key Stage 2 Level||Percentage achieving 5+ A*-C inc. E&M|
*From 2015 KS2-4 transition matrix for AC5EM. Source: RAISEonline transition matrices
The table shows that the large patch of orange and yellow in the middle of the graph - schools with prior attainment equivalent to a 3a or 4b, which make up the majority of schools between them - are not destined for high AC5EM passrates. Only 13% and 25% of students with 3a and 4b respectively at KS2 will attain this threshold. It’s therefore not surprising that our earlier table shows schools with an average prior attainment of 3a and 4b who are achieving expected progress can have AC5EM passrates as low as 8% and 34% respectively.
The skewed perception of a school that can be made by looking at AC5EM is exemplified by St Thomas More Catholic School in Haringey, North London. If we just look at this school’s AC5EM passrate then it looks decidedly average: at 57%, it is equal to the national average. It’s Value Added score, however, is 1,066.4. This makes it the 6th best school in the country (amongst those for which we have data) when it comes to student progress. Once you realise that the students of this school have average prior attainment equal to a 3a, it’s hardly surprising that reaching a national average AC5EM has consisted of such spectacular progress. If you look at the table above, you will see that only 13% of students with this KS2 level will attain the AC5EM threshold nationally.
We’ve chosen AC5EM to illustrate the issue with attainment measures and our consequent preference for progress measures, but even if we use an average rather than a threshold measure (and we explain below why this is preferable), the basic problem persists: knowing that a school has high or low attainment doesn’t give you much of a clue about whether this figure represents decent attainment for the students in the school or not.
Why We Prioritise Averages Over Thresholds
As explained above, one problem with threshold measures is that the bar they set a bar of equal height across all schools, which may represent a difficult or easy target depending on the school.
However, there’s also a second problem: instead of being simple ‘in’/’out’ measures, threshold measures such as AC5EM and Ebacc require you to be ‘in’ on a number of different areas. Just slipping up on one - e.g. getting a D in your maths GCSE - means you are not counted in the bucket. Students who trip up on one area (e.g. D in maths but As and Bs in 8 other GCSEs including English) are treated the same by the measure as students who have low attainment across the board (e.g. 4 GCSEs at grade D). Equally, students who just scrape the threshold are treated the same as those who excel far beyond it. The more buckets you are required to perform across, the more opportunity there is for slipping up in one area. Increasing the likelihood for slip-ups introduces an element of randomness and uncertainty, and increases the likelihood of a school’s performance on the measure being skewed by the number of students who just missed the target in one area.
Perhaps this is the reason why there is a closer correlation between Best 8 APS per Pupil and Best 8 Value Added than there is between AC5EM and Best 8 Value Added (the former has an R-Squared of 67%, compared to the latter’s 53%).
Best 8 APS per Pupil against Best 8 Value Added
Average attainment measures such as Attainment 8 and Best 8 APS avoid these problems because they are able to tell you how well students did on average, which is particularly useful for understanding the attainment of schools in which AC5EM is either a very low or a very high bar.
That said, threshold attainment measures do still have their place, and can contribute usefully to the overall picture of school performance. Although we haven’t included AC5EM (which will in any case be phased out from next year), we have included Ebacc, which will continue to be one of the government’s accountability criteria for the foreseeable future. While it is a similarly-structured measure to AC5EM (i.e. a threshold across multiple categories), we think it represents a more meaningful level of attainment, and is therefore one of the things that should be taken into account when assessing a school.
Aiming for all students to attain a threshold level - and not writing off those who are starting from behind as unable to get there - is a noble aim. However, it should not be the primary means of assessing school performance. We’ve therefore prioritised those measures that we think do provide the best first impression, and deprioritised those which on their own can lead to flawed conclusions.