The Stern Review on REF 2014 – a review of recommendations

Following the recent publishing of the Stern Review of REF 2014, Dr Sergey Popov looks at some of the recommendations contained in the report.

Lord Nicholas Stern, the president of the British Academy, has prepared an independent review of the Research Excellence Framework (REF). The report, which was commissioned by the Minister of Universities and Science, addresses the outcome of the 2014 REF exercise, which judged the research output of all UK universities over the period 2008-2014 for the purpose of allocating government research funding. The scope of the report was also to examine the purpose and benefits of REF, identify any problems and set out a vision for the future of REF.

Stern finds that it was a costly exercise (£246 million), with rules sometimes distorted in spirit by the participants. Every full-time member of a university department needed to submit four published papers that were used to quantify the relative quality of the department based on the average score of individual members. REF also assessed research impact and environment.

The Stern Report identified two specific ways of distorting or “gaming “the system (page 12, 28-32):

1. Some institutions decided not to include all their faculty members in the REF submission, which improved the average score of submitters and lowered the number of necessary impact case studies to add to the REF submission.

2. Some schools were hiring people on a fractional contract basis from abroad (on a sabbatical, retirement, or otherwise temporarily available) before the REF for a year, then parting ways after including them in the REF submission as if they were a legitimate member of the department.

To combat these attempts, the report suggests a number of changes to the current REF system. Unfortunately, some of the recommendations provide more opportunities for gaming the system by manipulating the paperwork, not less.

Recommendation 3 states that “Outputs should not be portable”. This is to make headhunting before REF pointless: the papers of productive researchers will remain with institutions from where they were poached. This might have made sense if the UK did not have any international movement of faculty. However, if one realises that a productive faculty member could be leaving the UK entirely for another part of the world, e.g. the USA, then the original UK institution is recommended to be remunerated with higher rankings for the emigrant’s achievements (paragraph 73). Now, this would be the remuneration in years 2021-2026, years when the emigrant is definitely not in the UK. Can we really say that the REF results based on emigrants’ achievements from the previous decade somehow measure the current research environment in a given school?

To make things worse, this recommendation is supposed to remove the incentive to fake hire the gastarbeiter researchers from other schools before the REF submission moment. With the provision to keep the accepted outputs in the abandoned institutions, this recommendation creates new incentives for fake hiring every year, not only in the years before the REF submission moment. Indeed, if one hires a US-based prominent researcher now for the researcher’s sabbatical year, with Stern’s recommendations accepted as they are, this US-based researcher can go back to the US next year, and three of his/her papers would improve the REF rankings of the fake hiring UK school in 2021!

Recommendation 1 suggests, “All research active staff should be returned in the REF”. This recommendation is supposed to make the research intensity more transparent. Instead, it makes hiring new faculty members very costly, especially combined with Recommendation 3. It creates an incentive to hire people for one year, dismiss them next year (for instance, for not winning a big enough grant, where big enough is never specified explicitly), and capitalise on papers published while these temporary researchers are on campus, averaging them over incumbent faculty.

With a crisis in academic circles in Turkey, and an ample supply of freshly minted PhD programme graduates from within the UK, this is perfectly sustainable until 2021 – but it is not clear if adopting such a practice equals creating an adequate research environment. This is exacerbated by paragraph 67 of the report that motivates Recommendation 2 on the flexibility of REF submissions: individual researchers may no longer need to submit a single research output; instead any number between 0 and 6 is acceptable, as long as the average number of submissions is two in the department. A zombie department where by 2021 faculty published zero papers altogether but they file a REF report that contains only papers by their former colleagues would become perfectly plausible.

There are many other problems with individual academic incentives (why would one work on more than two papers per six years if one needs two papers to not worsen the average numericals?), academic dishonesty (such as procrastinating with resubmission of a revised paper before a move), and so on. But even assuming that the changes proposed by the Stern report will not affect individual decision-making by researchers, they not only fail to remove the rent-seeking motivations present in REF 2014, but they also clearly allow much worse rent-seeking opportunities for the university administrators.

Ignoring the chance that the subjects of the REF process can respond to the opportunities provided by the recommendations would be fine if the report was not dealing with UK university professors and administrators, who, based upon the high praise in the Stern’s introduction to the report, are likely to be quite smart!

The views expressed in this article are the author’s own and do not reflect those of the institution.

 

The featured image in this article has been used thanks to a Creative Commons licence.