Realising Registered Reports Part 3: Dispelling some myths and addressing some FAQs

While collecting more signatures for our Registered Reports campaign, I spent some time arguing with people on the internet – something that I don’t normally do. Internet arguments, as well as most arguments in real life, tend to be about people with opposing views shouting at each other, then going away without changing their opinions in the slightest. I’m no exception: nothing of what people on the internet told me so far convinced me that registered reports are a bad idea. But as their arguments kept repeating, I decided to write a blog post with my counter-arguments. I hope that laying out my arguments and counter-arguments in a systematic manner will be more effective in showing what exactly we want to achieve, and why we think that registered reports are the best way to reach these goals. And, if nothing else, I can refer people on the internet to this blog post rather than reiterating my points in the future.

What do we actually want?

For a registered report, the authors write the introduction and methods section, as well as a detailed analysis plan, and submit the report to a journal, where it undergoes peer review. Once the reviewers are happy with the proposed methods (i.e., they think the methods give the authors the biggest chance to obtain interpretable results), the authors get conditional acceptance, and can go ahead with data collection. The paper will be published regardless of the results. Registered Reports is not synonymous with Preregistration: For the latter, you also write a report before collecting the data, but rather than submitting it to a journal for peer review, you upload a time-stamped, non-modifiable version of the report, and include a link to it in the final report.

The registered reports format is not suited for all studies, and at this point, it is worth stressing that we don’t want to ban the publishing of any study which is not pre-registered. This is an important point, because it is my impression that at least 90% of the criticisms of registered reports assume that we want them to be the only publication format. So, once again: We merely want journals to offer the possibility to submit registered reports, alongside with the traditional formats that they already offer. Exploratory studies or well-powered multi-experiment studies offer a lot of insights, and banning them and publishing only registered reports would be a very stupid idea. Among the supporters of registered reports that I’ve met, I don’t think a single one would disagree with me on this account.

Having said this, I think there are studies for which the majority of people (scientists and the general public alike) would really prefer for registered reports and pre-registration to be the norm. I guess that very few people would be happy to take some medication, when the only study supporting its effectiveness is a non-registered trial conducted by the pharmaceutical company that sells this drug, and shows that it totally works and has absolutely no side effects. In my research area, some researchers (or pseudo-researchers) occasionally produce a “cure” for dyslexia, for example in the form of a video game that supposedly trains some cognitive skill that is supposedly deficient in dyslexia. This miracle cure then gets sold for a lot of money, thus taking away both time and money from children with dyslexia and their parents. For studies showing the effectiveness of such “cures”, I think it would be justifiable, from the side of the parents, as well as the tax payers who fund the research, to demand that such studies are pre-registered.

To reiterate: Registered reports should absolutely not be the only publication format. But when it comes to making important decisions based on research, we should be as sure as possible that the research is replicable. In my ideal world, one should conduct a registered study before marketing a product or basing a policy on a result.

Aren’t there better ways to achieve a sound and reproducible psychological science?

Introducing the possibility of publishing registered reports also by no means suggests that other ways of addressing the replication crisis are unnecessary. Of course, it’s important that researchers understand methodological and statistical issues that can lead to a result being non-replicable or irreproducible. Open data and analysis scripts are also very important. If researchers treated a single experiment as a brick in the wall rather than a source of the absolute truth, if there was no selective publication of positive results, if there was transparency and data sharing, and researchers would conduct incremental studies that would be regularly synthesised in the form of meta-analyses, registered reports might indeed be unnecessary. But until we achieve such a utopia, registered reports are arguably the fastest way to work towards a replicable and reproducible science. Perhaps I’m wrong here: I’m more than happy to be convinced that I am. If there is a more efficient way to reduce publication bias, p-hacking, and HARKing, I will reduce the amount of time I spend pushing for registered reports, and start supporting this more efficient method instead.

Don’t registered reports take away the power from the authors?

As an author, you submit your study before you even collect the data. Some people might perceive it as unpleasant to get their study torn to pieces by reviewers before it’s even finished. Others worry that reviewers and editors get more power to stop studies that they don’t want to see published. The former concern, I guess, is a matter of opinion. As far as I’m concerned, the more feedback I get before I collect data, the more likely it will be that any flaws in the study will be picked up. Yes, I don’t love being told that I designed a stupid study, but I hate even more when I conduct an experiment only to be told (or realise myself) that it’s totally worthless because I overlooked some methodological issue. Other people may be more confident in their ability to design a good study, which is fine: They don’t have to submit their studies as registered reports if they don’t want to.

As for the second point: Imagine that you submit a registered report, and the editor or one of the reviewers says: “This study should not be conducted unless the authors use my/my friend’s measurement of X.” In this way, the argument goes, the editor has the power to influence not only what kind of studies get published, but even what kind of studies are being conducted. Except, if many journals publish registered reports, the authors can simply submit their registered report to a different journal, if they think that the reviewer’s or editor’s request is driven by politics rather than scientific considerations. This is why I’m trying to encourage as many journals as possible to start offering registered reports.

Besides, if we compare this to the current situation, I would argue that the power that editors and reviewers have would either diminish or stay the same. It doesn’t matter how many studies get conducted, if (as in the current system) many of them get rejected on the basis of “they don’t fit my/my friend’s pet theory”. Let’s say I want to test somebody’s pet theory, or replicate some important result. In my experience, original authors genuinely believe in their work: chances are, they will be supportive during the planning stage of the experiment. Things might look different if the results are already in, and the theory is not supported: then, they often try to find any reason to convince the editor that the study is flawed and that the authors are incompetent.

As an example, let’s imagine the worst case scenario: You want to replicate Professor X’s ground-breaking study, but you don’t know that Professor X actually fabricated the data. It’s in Professor X’s interest to prevent any replication of this original study, because it would likely show that it’s not replicable. As the replicator, you can submit a registered report to journal after journal after journal, and it is likely that Professor X will be asked to review it. Sure, it’s annoying, but at some stage you’re likely to either find an editor who looks through Professor X’s strategy, or Professor X will be too busy to review. Either way, I don’t see how this is different from the usual game of get-your-study-published-in-the-highest-possible-IF-journal that we all know and love in the traditional publication system.

And, if you really can’t get your registered report accepted and you think it’s for political reasons, you can still conduct the study. I will bet that the final version of the study, with the data and conclusion, will be even more difficult to publish than the registered report. But at least you’ll be able to publish it as a preprint, which would be an extremely valuable addition to the literature.

I’m still not convinced.

That’s fine – we can agree to disagree. I would be very happy if you could add any further arguments against registered reports in the comments section below, that cannot be countered by the following points:

(1) Other publication formats should continue to be available, even if registered reports become the norm for some types of studies, and

(2) Definitely, we need to continue working on other ways to improve the replication crisis, such as improving statistics courses for undergraduate and graduate psychology programs.

I think that registered reports are a good idea. What can I do to support registered reports?*

I’m very happy to hear that! See here (about writing letters to editors) and here (being a signatory on an open letter and letters to editors). 

-------------------------------------------------------

* I wish this was a FAQ...

Edit (1.8.2019): In response to a comment, I've added a sentence about preregistration, and how it differs from RRs.