I generally roll my eyes whenever someone prophesises an upcoming apocalypse. I’ve heard enough promises of fire and brimstone that have not yet come to pass to remain optimistic about most things. Yet, when I was holding my annual open science workshop for a masters programme at my local university, I found myself telling the students: “I predict that the peer review system will collapse in the coming five years.” Now, I’m not sure if I’m right about this: Although I’ve heard many apocalyptic predictions throughout my life, none of them has actually become reality, so chances are pretty high that I’m wrong.
I’ve been meaning to write a blogpost about this for a while, but I’m not sure there’s anything new to say. After all, complaining about peer review is one of the favourite pastimes of academics. The trigger for writing it was a provocative post on bluesky by Samuel Mehr. He wrote: “my reactionary take is that peer review is fine, on balance, and most people who hate it feel this way for reasons that are tangential to a real assessment of peer review (eg, they're annoyed about how a specific case of peer review played out, probably one that didn't go their way)”. Although I responded, this seems like a difficult topic to discuss on social media without mixing up different issues. So, here is a blogpost attempting to summarise the issues and the reasons for my prediction. The problem is: I don’t have a solution for the issues that I see with peer review. But it seems to me that there are options that are at the very least not worse than what we currently have.
Issue 1: People hate peer review because they have bad experiences where their own papers were rejected
Here, I’ve paraphrased Samuel’s main point from his bluesky post, but I’d like to discuss it from a different perspective. I’m sure that he’s right that many people hate the peer review process partly (and perhaps subconsciously) because the reviewers fail to see the genius behind their amazing papers. Of course – let’s face it – our article is probably less ground-breaking and more flawed than we’d like to think. Especially as an early career researcher, it can be disheartening to submit a paper that consists of your blood, sweat and tears, and that you’re convinced will change the world, only to have it ripped to shreds by the reviewers. At least in my experience, it’s a learning process to realise that your work is nowhere near as important to other people as it is to you.
This is an important realisation, but the question is what is the best way to find this out. At this stage, I have to second Samuel’s observation that most of the reviewers are thoughtful, helpful, and show a high level of competence. Learning how to discuss with people who completely and utterly disagree with me has been a fun challenge for me over the last decade, and it involved learning to take my own work less seriously, on a personal level. Arguably, the peer review process, as it currently stands, is a good way to acquire this skill.
Conversely, one might argue that it’s not: if this many people hate the peer review process because it hurt their ego, we should at least consider whether there is a way to teach these skills that may be a little less cruel. In my experience, there may be, for every 20 thoughtful and kind reviewers, one with a broken caps lock key and questionable manners. It would be bad luck, but not exceptional, to encounter such reviewers for the very first paper one submits. Should we protect authors from the caps lock guy? I think we should, because the peer review would work much better if we move it away from antagonistic feelings and towards fruitful discussions. I’m not sure about the best way to do this, though.
Publishing the reviews alongside an article could be an option: the caps lock guy may be more careful with their wording when they know the general public can actually read their review. Although one may also consider posting the reviewer’s name along with their review, I wouldn’t make this mandatory: the peer review process needs to be such that an early career researcher wouldn’t be intimidated if they want to point out some fundamental flaw in a big shot’s paper. The empirical question is if caps lock guy is confident enough to post a caps lock review publicly, if it’s anonymous.
An alternative way to tame the caps lock guy would be on the level of journals and editors: they could include a statement that they discourage bad manners and ask the reviewers to provide the type of review that they would have liked to have received back in their days as an early career researcher. This may work as a gentle nudge, but it would be difficult to impose any rules. If editors get the power to decide whether they will forward a review to the authors and use it to inform their decision, based on the review’s tone, this may introduce additional biases in the peer review system (e.g., “You were mean to my friend!” vs. “You were mean to a person with whom I have a long-standing professional rivalry!”). Furthermore, let’s say an editor does decide to reject a review based on the tone. Then, they would need to go out and recruit an additional reviewer, which would delay the publication process and may make the editor’s job even more difficult. To confound this issue, reviewers may be more reluctant to agree to review a paper in the first place, if they are worried that the reviewer may reject their review because they don’t like the tone. Although we might think that we know a non-constructive and rude review when we see one, there are also cultural differences in how directly one expresses criticism. In other words, German and Dutch reviewers may be barred from reviewing any papers for American journals if editors can reject the reviews based on the tone.
So, nicifying the peer review process would be good, but there’s no straight-forward way to achieve this. At the same time, in the spirit of remaining sceptical about my own scepticism about peer review, I should note that this is not an issue that is directly and intrinsically related to the concept of peer review: provided that peer review can be, theoretically, an open and constructive dialogue, this first issue is not an argument against the whole concept of peer review.
Issue 2: Lack of transparency and quality control
The second issue has been debated a lot, so there’s not much to say about that. The peer review process happens behind closed doors. This means, firstly, that a lot of academics get little acknowledgement of their behind-the-scenes editorial and reviewing work. And secondly, there is a lack of transparency about how the decision to publish a paper – or reject it – has been reached. The is especially important for the latter scenario: it’s probably a good thing that not every paper that was ever written will see the light of day, but at the same time, we don’t know how many good papers have gotten thrown out because of a discouraging but unfair initial review.
For published papers, the solution is relatively simple: Publish the reviews. Some journals have started doing this, as was pointed out to me in the bluesky thread I linked to above. Publishing the reviews will also allow any reader to get an idea of the quality of the reviews. Importantly, it will allow meta-scientists to compare the quality of reviews across different formats of peer-review. (I was going to enrich this blog post with some analyses like this, but I decided that it would be too time consuming to scrape and analyse the amount of reviews that would be needed to reach any meaningful conclusions.)
We have to be careful not to conflate issues, though. A lack of transparency is not an integral aspect of peer review. In fact, we can easily move towards open peer review. The problem would remain, however, that we would never see the reviews of papers that don’t get published.
Issue 3: Peer review as career gate keeping
During and shortly after my PhD, I was sad about rejections because they hurt my ego. As I became more confident in myself and (rightly) less confident in the ingenuity of my papers, I stopped being sad about this, but I gradually became more sad because every rejection means a smaller chance of grants and positions, which is strongly linked to an existential threat of having to leave academia. I consider myself lucky, because I have always worked in labs where my advisors and colleagues valued quality over quantity, and there was no formal pressure to publish. For example, I did not have any rules about the number of papers that I need to publish in order to get a PhD. My current PhD students do have such a rule, imposed by the university: Two accepted articles in a high-IF journal, at least one as a first author. Such rules are stupid and harmful (as an aside: I don’t think that, as a reviewer, I’d always pass a tone check by the editor).
First, the pressure to publish is stupid, because it’s not necessarily an indicator of a researcher’s worth. This has been debated to no end already, so just to add my two cents: Maybe there is some positive correlation between quantity and quality of publications. However, in my view, the number of deviations from this regression line are sufficient to make quantity useless as an overall indicator: for example, when people are pressured to salami-slice instead of producing an in-depth and coherent piece of work, or when bullshitters get tenure at the cost of more competent people leaving academia.
Second, the pressure to publish is harmful, because it creates the wrong incentives. The increase in the number of papers is exponential. Nobody has the time to read all these papers, and nobody has the time to review all these papers. On bluesky, Brad Wyble replied that the difficulty with getting reviewers is, empirically speaking, not as bad as is often assumed. However, in my experience, both as an editor and as the recipient of desperate-sounding invitations to review papers, I’d say it’s pretty bad. Needless to say, if editors are just happy to find anyone to review a paper, one can’t expect that all reviewers will have the in-depth knowledge and expertise that one might hope for.
It's not an encouraging thought for a researcher that they are producing papers that very few people actually have the time to read, and that will thus not even lead to an incremental progress in our understanding of their phenomenon of interest. But a substantially more harmful effect of the pressure to publish are the publications by paper mills. I won’t write too much about them, because other people who are much more knowledgeable about this topic, like Dorothy Bishop, have. But, in short, if you pressure people to write papers, they will write papers, leading to an inflation of all kinds of publications, including such that are useless at best.
How should we evaluate researchers, if not by the number of publications or related quantitative measures? Frankly, I have no idea. In answering this question, therefore, I will cheat by saying that this is an orthogonal issue that is beyond the scope of this blogpost.
To be fair to the concept of peer review, the issue here is not peer review, but the incentive structures that have been built around it.
Issue 4: Incentivising reviewers and editors
I’m sure most of us have had the experience of reading a published paper and thinking: “How on earth did this get past peer review?” The answer to that question can often be obtained by looking at the names of the editor and author, and checking, at the next conference, whether the two of them hang out with each other. At other times, the reviews are just bad: maybe there is a conflict of interest, maybe the reviewers didn’t get the point of the study or didn’t have the knowledge to identify some fundamental flaw, and maybe the reviewers just didn’t care and waved through the paper so they could move on to the next task. Or they didn’t want to ruffle any feathers – even in anonymous peer review, one can often guess who the reviewer is, and, you know, I scratch your back, you scratch mine.
How do we get high-quality reviews? How do we counteract the inevitable biases that exist both on the side of the editors and of the reviewers? How do we incentivise the reviewers to provide a detailed and honest report? Perhaps these are empirical questions – if we have open peer review, and can systematically investigate which factors affect the quality of the reviews. Again, this is not a killer argument against the whole concept of peer review. However, these are open questions, and the system, as it currently is, is definitely far away from some optimal solution.
Conclusion and ways forward
It’s easy to get side tracked in the debate about peer review, so let’s get it back to the basic questions: Does peer review, as it currently is, do its job? In terms of doing its job as a quality indicator of articles, I’d say: Kind of. In terms of determining which researchers deserve to get jobs and grants, I’d say: Fuck no. And in terms of a decision mechanism about which articles should be published and which ones should be stowed away in a file drawer forever, I’d say: no, not really, no.
Still, we should be careful not to conflate issues, and not to throw out the baby with the bathwater, as the saying goes. Ceteris paribus, a paper with positive peer reviews is likely to be better than a paper with negative peer reviews. Many of the issues are not intrinsic to the process of peer review per se and would not be impossible to change. Specifically, I refer to the lack of transparency and the incentive structure.
Transparency can easily be increased by publishing reviews with a paper. This leaves us with the problem, however, that we will never see the reviews of papers that get rejected. A way to fix this would be to publish preprints. Technically, it shouldn’t be difficult to allow for anyone to write public reviews for any preprint, thereby breaking the monopoly of academic journals to provide peer review. Even a thumbs up / thumbs down button next to a preprint could provide some indicator about whether a paper should be taken seriously or not. The problem that remains are all kinds of biases: For example, someone who is new to a field and not well-connected in the community might find it difficult to get any kind of reaction for their preprint. This might lead to a cycle where their paper won’t be noticed by the community because it’s not reviewed, and won’t get reviewed because it has gone unnoticed. This is why I like the concept of Peer Community In …, as it is a way to attach peer reviews to a paper when uploading a preprint.
And as for the incentive structure, there’s very little to say, aside from arm waving and prophecies of an apocalypse. It ought to be changed. But nobody seems to know how.
So, will peer review collapse in the coming years? Maybe not. Despite what it sometimes feels like, things are changing. Open peer review, a wide spread of preprints, and platforms such as PubPeer and PCI have only appeared in the past decade or so. Not everyone is convinced about these novelties, but they somehow co-exist with the traditional closed peer review system by for-profit journals. Perhaps the goal should not be to topple the existing system, but to further develop alternatives and experiment with what yields the best results. And hope for the best.
