ARQM: Astonishingly Reckless Quality Monitoring

Our research ‘quality’ is being assessed by the ARQM in a way that is astonishingly reckless. The ARQM requires every ‘research active’ member of staff to submit their ‘best’ four publications from the last four years. These are now being rated and assigned a score (supposedly equivalent to the REF star ratings) and each member of staff will receive an ‘average’ between 0 and 4.

There are many problems with the ARQM, including:

  • Even the REF panels do not provide marks for individuals because they acknowledge that at the individual level scores are not reliable. It is only the aggregate score in which they even claim to have confidence. The ARQM therefore claims a level of accuracy that the (much criticised [1, 2, 3], but nonetheless much more rigorous) REF process didn’t claim.
  • During REF two experts, usually from a relevant disciplinary sub-field, review your publications. For ARQM one person (a ‘senior member of staff’) often entirely outside of your field or unversed in your research design or methods, assesses your work. They are being asked to make multiple assessments very quickly and not being granted additional support to do this. The validity of any assessments must therefore be extremely doubtful, something that we know has been raised by many senior staff involved in the rating process.
  • A rolling four year average is used. This means that if you published three great things in 2012 you only need one additional piece since then to have a 4* ARQM rating for the next two years. If you haven’t published sufficient new 4* pieces by 2015, you may then plummet to 1* or 2*. This variability makes no sense; we don’t suddenly become, or stop being, internationally excellent. Research is a process with publication peaks and troughs because research is cyclical. Moreover, so long as we publish four strong things within a REF cycle we will contribute to the University’s research excellence. Because we know this, many of us who already have four good publications, use the later years of a REF cycle to publish things we think are socially useful, even ‘impactful’, but not perhaps very REF-able (reports, chapters, reviews of the field, textbooks). The imposition of ARQM, with its endlessly rolling cycle, does not match the REF cycle, does not allow for lulls and punishes us for being good academic citizens.

ARQM not only produces a score without validity. It is also increasingly being employed in astonishingly reckless ways across the university. To put it bluntly, your ARQM is affecting your job. Already, it is being used, without consultation with the UCU or (as far as we can tell) ratification through Senate, as a means of determining:

  • Promotions and progression;
  • The hours of teaching staff do – workload distribution;
  • The ability of staff to apply for sabbatical leave;
  • The eligibility of staff to supervise PhD students.

Even if the ARQM accurately measures research outputs over the last four years (which as we note above is extremely doubtful), there is no reason to think that staff research will improve if those who performed at a lower level over the last few years now do more teaching and have less research time. Nor will people who score highly continue to prosper if they are required to supervise all our research students (and of course there is no reason to believe that a higher ARQM score makes you a better supervisor). Surely it is in the University’s and our interests if all staff experience the conditions to produce better research?

In short, the ARQM is a worthless process that is being used to remove discretion from departments and schools under a false guise of transparency. It has no place in academic decision making.

Leave a Reply

Your email address will not be published. Required fields are marked *