The Guardian, League tables, and the Strategic Incentive to Inflate grades

I really like the Guardian. It is one of the world’s great newspapers. They do excellent work and I genuinely believe the world is a better place because of the Guardian, but I do not agree with everything they do.

Let’s talk about grade inflation. Let’s talk about league tables. The UK’s big three may be found here, here, and here. I am going to pick on the Guardian League Table because it is to hand and they have a column index called “value added” score. 

What is “value added”? As a foreigner who has lived in the UK for close to 8 years, it sounds like something that should be taxed, as in value added tax (VAT, the UK equivalent to a sales tax). This isn’t what added value means here.

What is it then? Value added is an index related to degrees awarded at a particular classification against the likelihood of getting such a degree based on a set of criteria such as entry qualification. A description of the methodology is here. The worrisome part is this:

 

"Thus an institution that is adept at taking in students with low entry qualifications, which are generally more difficult to convert into a 1st or 2:1, will score highly in the value-added measure if the number of students getting a 1st or 2:1 exceeds expectations."

Let’s be cynical for a moment. League tables matter. Suppose you are a strategist at a university who is rewarded for meeting league table position targets. One easy way to improve your position is to double the number of 2.1 and first class degrees. This would lead to instant success, unless the others do it faster than you. On the other hand, suppose you maintain your standards over the long term and are competing against others prepared to lower standards, what do you think will happen? Your institution will sink like a rock. 

This is rather cynical and would never happen, would it? 

Yes it would. Let’s be clear. There has been massive grade inflation in UK Universities. If you need evidence, it is here and here. For those interested in longer term trends there is some interesting data here (the useful part begins on page 22). I am not going to blame the current government and its policies, or the last government and its policies. Grade inflation is a long term problem and Universities are punished for not going with the FLow. 

How are they punished? They are punished in league tables by indexes such as “value added”. They are punished by employers who, justifiably, have decided that in general a 2.1 degree isn’t what it used to be and a 2.2 degree is worse than it used to be and hiring students with grade inflated degrees preferentially. After not having their graduates hired in preference for those with inflated degrees, they are punished a second time in the league tables based on graduate employability.

Although I am picking on the Guardian, they are not alone. They are one of many. League tables consider “good honours degrees” and “completion rates” not to mention “spend on student facilities” and various indicators of “satisfaction”. Yes these are important considerations, but as drivers of University behaviour over the long term I think the sum is far worse than the individual parts. Overall, the broad enterprise encompassing the league table generators and the strategists at individual universities have much to answer for. 

Does it matter? Yes it does. Enormous top down pressure can be exerted for the purpose of changing league table position rather than improve education.

Note: For those interested in a deeper look beyond league tables as well as commentary about predictors of league table position and university strategic decisions to maximise position, the US National Academy Press published this report on “Improving Measurement of Productivity in Higher Education”. Parts of particular relevance to this discussion are on pages 53 (strategic decisions) and 70 (popular proxies for league table position). The issue of proxies is particularly relevant in that 79% of the variance in the Guardian’s university league table can be explained by a very simple logarithmic model based on entry Tariff (US readers this is based on standardised exams). The data below are from here as it appeared on June 18, 2013.

 

 

Statement relating to possible conflict of interest and a comment: The Guardian and others rank my university and the subject I primarily work in (Chemistry). At various times we have been positioned more or less well. I am concerned about our place in the tables because, in the current climate, these things matter. I like to think my concern is independent of this and that I would say the same thing no matter the position of my university in the tables. I will leave the reader to judge. It also makes me better appreciate the bravery of my UG institution for refusing to participate in league tables.