League Tables and the Bias of the Neanderthal Brain

The Silly Season continues in Universities as the talking points and marketing material are generated. Leagues table updates are out and everyone rushes to interpret the last data point. The disease propagates from the League table publishers to the universities, the government, the media and Mr and Mrs Somewhat-above-average. What is so silly? That the last data point is interpreted at the expense of longer term better data sets. If you do this, remember...

The Excursions May Be Great but Mostly They Stay the Same. Looking again at the data set of 5 selected Universities (Derby, Nottingham, Nottingham Trent, Sheffield and Sheffield Hallam) it is worth considering the 5 year average, the median, and the standard deviation as well. In all cases the median values are very close to the averages and while there is some skew (asymmetry) to the data mostly they are just randomly fluctuating about a mean.

University	Average League Table Position	Standard Deviation	Median
Derby	100	8.1	103
Nottingham	20	3.1	19
Nottingham Trent	56	5.6	55
Sheffield	26	1.1	26
Sheffield Hallam	69	5.4	68

 If you are an academic manager celebrating a single year rise in league table position, you are probably celebrating random fluctuations. If that is what you are into and that makes you feel better about yourself, well... with all due respect, you are an idiot.

As a student, if you make a decision based on a University showing a large one year jump in league table position, you may be disappointed in subsequent years. That university may drop back in following years. It is unfair that you are being marketed to in this way. As an Academic, I am sorry. The best I can do is encourage you to be careful and to point out the dark side to this behaviour. Universities using league table results to promote their wares do so with a bias befitting the Neanderthal brainstem. This systematic bias is unworthy a modern Institution of higher learning.

University marketing materials use this material ONLY when it is in their favour to do so. They do not give equal weight to downward results. If you listen, at best you are making a decision on a biased presentation of flimsy data because over 5 years or so, they are reasonably stable about a mean value. If they generally stay the same while you are earning your degree, you should ignore league tables and make your decision in other ways.

League tables: Performance today is not performance tomorrow.

It is silly season in Universities. It is the time of year when the league table updates come out and universities start interpreting that last data point. This is one of several posts on the disease visited upon UK higher education. Yes, I am talking to you at the guardian and you at the Complete University Guide. 

Performance today is not performance tomorrow. It takes 4-5 years from the time a student chooses a university and when they graduate. League table position varies over that time. Consider 5 universities: Derby, Nottingham, Nottingham Trent, Sheffield and Sheffield Hallam. These have been chosen because they are all close by and are within about 50 miles of each other.

University	Average League Table Position	Range	Range as percent of average
Derby	100	21	21
Nottingham	20	7	35
Nottingham Trent	56	13	23
Sheffield	26	3	11
Sheffield Hallam	69	14	20

These data mean that over 5 years a University’s league table position may vary greatly both in absolute and real terms. These can be very large even in a single year.  Northampton jumped up 39 places 2013-2014. The Royal Agricultural University dropped 32 places 2014-2015. Really. These changes are meaningless either way. Having seen the inside of many Universities, I can assure you real change mostly happens slowly and mostly no one notices. 

Take home message: If you are making a decision about your choice of University based on 20-30 league table positions, you are making a bad decision. If you are evaluating your institution based on a single year's upward or downward change in league table position, you are likely doing it badly.