The Band-aid on the Broken Leg: REF/RAE

      Looking through an email sent from jobs.ac.uk, an advert for 2 REF science writers placed by the University of Glasgow was notable. Each is for a 6 month fixed term contract at £31,948 - £35,938/yr, with a carrot of possible extension dangled in front of a prospective applicant. This served as a reminder, that there is something bothersome about the REF.

What is REF? For those outside the UK academic system, REF stands for Research Excellence Framework and is the latest incarnation of what was previously called the RAE (Research Assessment Exercise). The essence is that every few years universities compete for a pool of cash from HEFCE (The Higher Education Funding Council for England). The last RAE was 2008 and the REF2014 process is spurring these adverts. The idea is to assess the quality of research and its impact according to a series of criteria. The stakes are huge: the University of Oxford with a huge and great research enterprise might get £126 Million in a year (2010/11) while others might get less. The amount received is related to the assessed quality of the research and the scale of the enterprise (by how many staff are entered).

Before I make the following statement, I need to make something crystal clear: the opinions expressed here are NOT those of my employer.

That said, I think the REF process and administration of it are a tremendous waste of resources and the way it is done structurally flawed. To clarify, there is nothing wrong with assessing research. There is nothing wrong with stepping back and assessing the value of a public investment. This is not the issue.

What are the issues then?

1.  It promotes instability and inhibits long term decision making by having a large (in some cases) and volatile income source. 
2.  It interferes with the overall funding structure of science within the UK. 
3.  It inhibits bold measures to promote research growth within an RAE/REF period.
4. There are simpler ways of assessing. 
5.  It rewards the past rather than learning from the past to inform policy and funding decisions for the future. 
6.  It is hugely costly in terms of strategic and high level academic time from VC level down. The publicly visible cost represented by the two positions in the advert noted here are a tiny tip of the iceberg. 
7.  It interferes with individual academic publication decisions by focusing on journal status rather than direct impact.

What a mouthful and diatribe, eh? How can it really be this messed up. Before I explain, let me remind the reader – I have nothing against assessment of research and public investment. Really I don’t. If you read my other posts, you will find a broad interest in measures of success underpinned by metrics. You may disagree with the metrics chosen and with me – but if you think I have a philosophical problem with the concept of assessment you need to read more carefully.

The explanation may take a number of posts, but let’s start with the fundamentals. The RAE/REF is part of what is sometimes referred to as a dual support mechanism for research. The two legs of the dual support are: 1) research grants provided by Research Councils, Charities, and Business and 2) the REF/RAE. Why is “dual support needed”? To understand, typical “cost models” need to be explained.

To begin, let’s consider the interestingly named “Full Economic Cost (FEC)” model used by UK research councils for many grants. An example is here.  In this model a researcher applying for money computes the amounts needed over the period of the grant (salaries, stipends, travel, consumables, equipment, etc). This would be 100% of direct cost. To this is added an “overhead” which in my experience depends on the number of staff employed via the grant. This should cover estate, finance support, purchasing, electricity, and all those other things a researcher needs. The sum of the two (direct + indirect) in an ideal world should cover the full cost of the work done. This then is considered the FEC of the grant and the funding body will pay 80%. The actual computation is a little more complicated than this but for the purposes here is close enough. Where does the remaining 20% come from? One place might be the REF/RAE cash pool.

Let’s consider a typical charity grant. Charities often fund 100% of direct cost but seem to have an aversion to funding overheads. This is their prerogative; however, costs for things like buildings and admin support are real. Where does the money for this come from? One source might be the REF/RAE cash pool.

There is another model practiced by some charities and EU funding bodies called Additional Cost (AC). In this, a distinction is made between items like existing staff and staff hired for the purposes of the project. For example, a commitment of 20% of my time as a principal or co-investigator might be made toward the project. However, since I have a salary supported by the institution that cost doesn’t count as AC and the funding body doesn’t pay this. In the absence of that 20% commitment to the project, there are other things I am hired to do. So,... where does the money come from to cover 20% of a researcher’s other duties? One source might be the REF/RAE.

You get the picture. Against the status quo, imagine what the picture might be in the absence of that pool. Even with it, when last surveyed most (nearly all) chemistry departments were bleeding deficits attributable to research.

There is a problem here. S’pose as a home owner I decide I need a new roof. I get a quote and then tell the roofer that I will only pay for materials and the additional cost workers hired in for my job (but not the roofer’s time despite the fact he will actually be at my house working on the roof and directing the other workers), or I tell the roofer I will only pay 80% of the quote, or ... the roofer will likely teach me some new English slang I haven’t learned yet. The entire field of roofing would likely stagnate if the system was that the roofer had to enter a competition for a pool of government funds to fill the gap I created up to 5 years before. How do you create and grow new businesses in that environment. Similarly, once rewarded the business would still receive those funds for roofing, even if they no longer did any (at least until the next exercise).

What happens if an institution wants to rapidly grow research AND is able to bring in the grants successfully – deficit. Wouldn’t it be better to cut out the middlemen and move the funds over to to the funding councils so they can cover the cost of the work. It might also be sensible to do some seed funding of a super-charity whose job it is to top up charitable funds by 10-20%  to cover indirect costs.

That is where I will leave this post. I have touched on points 1, 2, and 3 above. See if you can see how points 4, 5, 6, and 7 might arise.

The impenetrable fog of tradition

An outsider viewing the US HE sector might have some basic, possibly penetrating, comments to make about the way it is organised. I cannot offer these as I am a product of that system, however, I am an outsider in the UK whose frame of reference is the American system, and, I do have some comments about two aspects of UKHE. Don’t get me wrong, I am not saying the American system is better (a quick look at some older posts should make clear an undercurrent of respect for the UK’s system overall), it is only a useful comparison.

The two items are A-levels and the Research Excellence Framework (REF). I will treat A-levels here and the REF later.

The A-level system is horribly messed up. The idea of standardised testing is not the issue – that is a topic on which adults may honourably disagree. Grade inflation is not the issue – this too is an issue on which citizens and government may debate. The range of topics at A-level is not the issue – there truly is a debate to be had about core subjects and modern topics. Let’s be clear, the horribly messed up part is not about the “what” of the A-levels system it is the “how.”

For the benefit of outsiders, let me briefly describe what happens...

In their final year of school, University bound UK students sit a set of exams called A-levels. My students might sit subjects like: Chemistry, Biology, and Business. Most of the subject areas you would recognise, others might leave you scratching your head. These exams provide a set of grades which Universities use to decide if a student can enrol on their preferred course of study in fall.

For North Americans (and others): The University year from a student perspective starts later in the UK than it does in North America. Usually orientation is the last week of September with classes beginning in October. August is mostly filled with family holidays provided by the generous annual leave packages provided by UK (and European employers).

Why are these points relevant? The test results are released in mid to late August. Prior to this (Decemberish), students apply to university based on a belief system reinforced by their teachers and whatever is going through their heads during their last year of school. They apply on the basis of “estimated” grades. The university “accepts” them based on this belief system. Universities track applications numbers, applications processed, etc... but at the end of the day what matters is the all important grades – as they should.

So far so good. The good part is that this system working well promotes ruthless self-assessment and reflection among 18 year olds... Or not...

 In an ideal world, they would choose a range of universities that match a range of possible outcomes, except when they don’t... Like the ambitious all A* (predicted) student who applies to Cambridge, Oxford, and Imperial and gets a B instead.

Reality bites and all the data I have seen suggests, for example, that the number of students who believe they will get net grades under 300 is smaller than the number who actually do.

But this is perhaps unfair because teachers are to blame too. They haven’t done a great job predicting and tend to over predict in ways that are biased by for gender, economic status, and institution type.

None of this would matter were it not for the fact that the grades are released so late. The results are obtained in the second half of August and a mad rush known as “clearing” begins. During the period, universities shed those who do not make the grades; students without a place (perhaps at all their choices) make a rush for the best alternative university. Some with excellent grades never find a place. 

The clearing season is complex because it should be a time when clarity finally appears. It is the first time that students and universities have the opportunity to abandon belief and take action based on facts.  Except they can’t always...

... because at this critical time, Universities often do not always know how many places they have left. Why don’t they know this? Students do not always convert to second choice institutions. There is another set of exams called BTECs, the results of which are released about the same time.Between the two it takes time to get clarity. Getting this sifted accurately takes about two weeks, sometimes longer. They may think they have no places one day and realise they have dozens the next. Courses get closed and a good student may not find a place. ...but if a very good student wants a place, why not accept them? Because the universities can't take them due to student number limits.

What is the problem with student numbers? Universities can be fined for taking too many students. During the horrible messed up period, the stakes are very high for both students and institutions. Worst of all, many decisions makers will be on summer holidays.

In conclusion, there is much good about examinations and trying to get reliable indicators of University success. But systems need to do this in a timely fashion so that students and universities can make critical decisions based on knowledge rather than belief. The data suggest the belief system represented by estimated grades is often wrong and gets more wrong as economic status decreases. The whole system would be aided by allowing all parties to have the data before applications are submitted. If one or two loose ends need to be tied up, get this done a few months in advance. But the problem is, there is some attachment to this last minute chaotic muddle. I have no idea why.