Wellcome peer review report

Copyright Wellcome Trust CC BY 2.0

Copyright Research Information Network

A Wellcome Trust commissioned report centred around the issue of peer review was published last month. The report, conducted by the Research Information Network, sets out a detailed analysis of peer review, the critiques, and the new alternative systems for peer review that have appeared in recent years.

The report defines traditional peer review as 'the process of subjecting an author’s scholarly manuscript to the scrutiny of others who are experts in the same field’ (p.6) These experts must assess whether the research is of sufficient quality to be included in the journal. Many journals also require reviewers to assess the originality and significance of research, however it is worth noting that some open access mega-journals such as PLOS ONE and Collabra do not require this.

The report details many of the criticisms commonly made of peer review. One criticism is that peer review is not effective at weeding out unsound research. The report makes mention of the fact that retraction rates have increased in recent years, although the volume of retractions is a very small proportion of the 2 million papers published each year globally. Another criticism mentioned in the report is the burden faced by reviewers, the vast majority of whom are unpaid. Research suggests that around 3 million papers are submitted for review each year, and many are submitted and reviewed more than once. This represents a significant burden on reviewers, half of whom spend more than 6 hours on each review (see our recent blog post on the new journal Collabra that compensates reviewers by paying them a proportion of APC income). Other criticisms include potential for bias, expense, delays, and the potential for subversive behaviour.

Given the criticisms and the potential new avenues afforded by new digital technologies there have been many experiments with alternative forms of peer review. To take one example, open review sees both author's and reviewer's names disclosed from the outset (for example BMJ and PeerJ) and is designed to encourage constructive comments and avoid overly harsh reviews. Journals such as Frontiers (see previous blog post here) have expanded on this idea and introduced an interactive communication element to their open review process.

After investigating the various forms of peer-review the report concludes that peer review "remains a bedrock of the scholarly communications system", but reflects that there is likely to be increasing pressure on traditional peer review systems as the rate and volume of scholarly communication increases. It is this continuing pressure that will ensure experimentation with new forms of peer review will continue with cooperation between research and publishing communities:

'[W]e detect a sense in which while publishers will continue to explore new approaches in the ways we have described, they would welcome more guidance from key sections of the research community on the kinds of peer review services they want from publishers, and on the purposes that they should seek to fulfil. Unless the purposes are defined with greater clarity than they are at present, at least some of the current experimentation may prove to be of little point.' Scholarly Communication and Peer Review: The Current Landscape and Future Trends, p.30. CC BY

The full report, Scholarly Communication and Peer Review: The Current Landscape and Future Trends, can be found here.