Accountability – What if we Can’t Actually Measure Pupil Progress?
Schools are now held accountable for how much progress their students make – but what if that’s something we can’t actually measure, asks Professor Becky Allen…
Measuring pupil progress has never been more important in schools. Accountability now rests on the Progress 8 score, so we find SLTs routinely monitoring whether each pupil stands a good chance of doing better than those students who had the same SATs score as them.
This is the only way they can predict whether their future performance table rating is secure.
However, measuring pupil progress in a subject, compared to similar learners in other schools, is incredibly difficult to do in practice, and is only possible where common assessments or moderation is used.
Recently, the number of schools making use of standardised tests in core subjects from commercial assessment companies or other organisations has increased. The idea is that a student takes one of these assessments each year so that the change in their attainment – ie their ‘progress’ – can be measured.
Uncertain judgements
It sounds so straightforward. However, research I have conducted on test data archived from Education Endowment Foundation trials has led me to conclude that these relatively short, standardised tests, designed to be administered in a 45-minute/one-hour lesson, are rarely going to be reliable enough to infer much about individual pupil progress.
No test can give us a perfect measure of exactly what a student knows in a subject. In measuring pupil progress, we are usually comparing test scores from assessments taken a year apart, where there is some necessary uncertainty about exactly what a young person was capable of doing at the start and at the end of the year.
This uncertainty means that, for most students, the amount of progress they make over the course of a year in a subject can rarely be described as statistically significant or noteworthy.
The problems with using these tests to measure pupil progress across schools go further than questions of test length and uncertainty in measurement.
Data from a single commercial test is unlikely to show whether a school is making good progress in a subject, unless we have standardised tests of a common curriculum, sat in standardised conditions, where students and teachers have standardised perceptions of the importance of the test.
The National Curriculum, even where it is followed, does not prescribe exactly what a school should teach each year, so the test materials rarely align to a school’s curriculum perfectly.
Furthermore, how much the test matters to pupils and staff in a school will vary considerably. For example, some teachers might know that these tests are used in their own performance management and so be highly motivated to push students to do as well as possible.
Some will be able to view the test papers in advance to influence revision guidance, and others not. There is an extensive academic literature that warns that how hard pupils try during a test to achieve a good mark will depend on whether they believe the test to be important, or not.
Of course, this too will vary across schools – depending on whether they use these standardised tests as formal end-of-year exams, whether they report results to parents, whether they use the test results to rearrange pupil class allocations, and so on.
Inconvenient truth
These concerns about achieving greater consistency in how tests are used in schools, whether through high quality guidance or other means, applies as much to multi-academy trusts devising their own tests as it does to schools that choose to buy commercial tests.
The difficulty is that for test score comparisons across schools to be valid, teachers need to know the circumstances under which they are being used in all other schools, not just their own.
This is why I am no longer convinced that any school can reliably measure pupil progress, compared to other schools. So, what should we do?
Accepting that we cannot reliably measure progress creates enormous problems for headteachers who want to hold teachers accountable for the progress of their classes, and for Ofsted inspectors who want to be assured that the school can ‘predict’ how well it will perform in the future.
I am convinced that one day, across many subjects, technology will allow the routine assessment of how much students are learning. But this is not where we are today.
And until then, we should take greater care in how we use this attainment data to report whether pupils are on track, and to hold teachers to account for progress made by their classes.
Professor Becky Allen is a Professor of Education at UCL Institute of Education. An expert in the analysis of large-scale datasets, last year she started the teacher survey tool, Teacher Tapp, and from 2015 to 2017 she was Director of Education Datalab.
