By how much would we need to increase our sample sizes to have adequate power with an alpha level of 0.005?

At our department seminar last week, the recent paper by Benjamin et al. on redefining statistical significance was brought up. In this paper, a large group of researchers argue that findings with a p-value close to 0.05 reflect only weak evidence for an effect. Thus, to claim a new discovery, the authors propose a stricter threshold, α = 0.005.

After hearing of this proposal, the immediate reaction in the seminar room was horror at some rough estimations of either the loss of power or increase in the required sample size that this would involve. I imagine that this reaction is rather standard among researchers, but from a quick scan of the “Redefine Statistical Significance” paper and four responses to the paper that I have found (“Why redefining statistical significance will not improve reproducibility and could make the replication crisis worse” by Crane, “Justify your alpha” by Lakens et al., “Abandon statistical significance” by McShane et al., and “Retract p < 0.005 and propose using JASP instead” by Perezgonzales & Frías-Navarro), there are no updated sample size estimates.

Required sample estimates for α = 0.05 α = 0.005 are very easy to calculate with g*power. So, here are the sample size estimates for achieving 80% power, for two-tailed independent-sample t-tests and four different effect sizes:

Alpha	N for d = 0.2	N for d = 0.4	N for d = 0.6	N for d = 0.8
0.05	788	200	90	52
0.005	1336	338	152	88

It is worth noting that most effects in psychology tend to be closer to the d = 0.2 end of the scale, and that most designs are nowadays more complicated than simple main effects in a between-subject comparison. More complex designs (e.g., when one is looking at an interaction) usually require even more participants.

The argument of Benjamin et al., that p-values close to 0.05 provide very weak evidence, is convincing. But their solution raises practical issues which should be considered. For some research questions, collecting a sample of 1336 participants could be achievable, for example by using online questionnaires instead of testing participants at the lab. For other research questions, collecting these kinds of samples is unimaginable. It’s not impossible, of course, but doing so would require a collective change in mindset, the research structure (e.g., investing more resources into a single project, providing longer-term contracts for early career researchers), and incentives (e.g., relaxing the requirement to have many first-author publications).

If we ignore peoples’ concerns about the practical issues associated with collecting this many participants, the Open Science movement may lose a great deal of supporters.

Can I end this blog post on a positive note? Well, there are some things we can do to make the numbers from the table above seem less scary. For example, we can use within-subject designs when possible. Things already start to look brighter: Using the same settings in g*power as above, but calculating the required sample size for “Difference between two dependent means”, we get the following:

Alpha	N for d = 0.2	N for d = 0.4	N for d = 0.6	N for d = 0.8
0.05	199	52	24	15
0.005	337	88	41	25

We could also pre-register our study, including the expected direction of a test, which would allow us to use a one-sided t-test. If we do this, in addition to using a within-subject design, we have:

Alpha	N for d = 0.2	N for d = 0.4	N for d = 0.6	N for d = 0.8
0.05	156	41	19	12
0.005	296	77	36	22

The bottom line is: A comprehensive solution to the replication crisis should address the practical issues associated with getting larger sample sizes.

Should early-career researchers make their own website?

TL;DR: Yes.

For a while, I have been thinking about whether or not to make my own website. I could see some advantages, but at the same time, I was wondering how it would be perceived. After all, I don’t think any of my superiors at work have their own website, so why should I?

To see what people actually think, I made a poll on Twitter. It received some attention and generated some interesting discussions and many supportive comments (you can read them directly by viewing the responses to the poll I linked above). In this blogpost, I would like to summarise the arguments that were brought up (they were mainly pro-website).

But first, and without any further ado, here are the results:

The results are pretty clear, so – here it is: https://www.xenia-schmalz.net/. It’s still a work-in-progress, so I would be happy to get any feedback!

It is noteworthy that there are some people who did think that it’s narcissistic for Early Career Researchers (ECRs) to create their own website. It would have been interesting to get some information about the demographics of these 5%, and their thoughts behind their vote. If you are an ECR who is weighing up the pros and cons of creating a website, then, as Leonid Schneider pointed out, you may want to think about whether you would want to positively impress someone who judges you for creating an online presence. Either way, I decided that the benefits outweigh any potential costs.

Several people have pointed out in response to the twitter poll that a website is only as narcissistic as you make it. This leads to the question: what comes off as narcissistic? I can imagine that there are many differences in opinion on this. Does one focus on one’s research only? Or include some fun facts about oneself? I decided to take the former approach, for the reason that people who google me are probably more interested in my research rather than my political opinion or to find out whether I’m a cat or a dog person.

In general, people who spend more time on self-promotion than on actually doing things that they brag they can do are not very popular. I would rather not self-promote at all than come off as someone with a head full of hot air. Ideally, I would want to let my work speak for itself and for colleagues to judge me based on the quality of my work. This, of course, requires that people can access my work – which is where the website comes in. Depending on how you design your website, this is literally what it is: A way for people to access your work, so they can make their own opinion about its quality. 

In principle, universities create websites for their employees. However, things can get complicated, especially for ECRs. ECRs often change affiliations, and sometimes go for months without having an official job. For example, I refer to myself as a “post-doc in transit”: My two-year post-doc contract at the University of Padova went until March last year, and I’m currently on a part-time short-term contract at the University of Munich until I will (hopefully) get my own funding. In the meantime, I don’t have a website at the University of Munich, only an out-of-date and incomplete website at the University of Padova, and a still-functioning and rather detailed and up-to-date website at the Centre of Cognition and its Disorders (where I did my PhD in 2011-2014; I’m still affiliated with the CCD as an associate investigator until this year, so probably this site will disappear or stop being updated rather soon). Several people pointed out, in the responses to my twitter poll, that they get a negative impression if they google a researcher and only find an incomplete university page: this may come across as laziness or not caring.

What kind of information should be available about an ECR? First, their current contact details. I somehow thought that my email address should be findable for everyone who looks for it, but come to think of it, I have had people contact me through researchgate or twitter and saying that they couldn’t find my email address.

Let’s suppose that Professor Awesome is looking to hire a post-doc, and has heard that you’re looking for a job and have all the skills that she needs. She might google you, only to find an out-dated university website with an email address that doesn’t work anymore, and in order to contact you, she would need to find you on researchgate (where she would probably need to have an account to contact you), or search for your recent publications, find one where you are a corresponding author, and hope that the listed email address is still valid. At some stage, Professor Awesome might give up and look up the contact details of another ECR who fits the job description.

Admittedly, I have never heard of anyone receiving a job offer via email out of the blue. But one can think of other situations where people might want to contact you with good news: Invitations to review, to become a journal editor, to participate in a symposium, to give a talk at someone else’s department, to collaborate, to give an interview about your research, or simply to discuss some aspects of your work. These things are very likely to increase your chances of getting a position in Professor Awesome’s lab. For me, it remains an open question whether having a website will actually result in any of these things, but I will report back in one year with my anecdotal data on this.

Second, having your own website (rather than relying on your university to create one for you) gives you more control of what people find out about you. In my case, a dry list of publications would probably not bring across my dedication to Open Science, which I see as a big part of my identity as a scientist.

Third, a website can be useful tool to link to your work: not just a list of publications, but also links to full texts, data, materials and analysis scripts. One can even link to unpublished work. In fact, this was one of my main goals while creating the website. In addition to a list of publications on the CV section, I included information about projects that I’m working on or that I have worked on in the past. This was a good reason to get myself organised. First, I sorted my studies by an overarching research question (which has helped me to figure out: What am I actually doing?). Then, for each study, I added a short description (which has helped me to figure out what I have achieved so far), and links to the full text, data and materials (which helped me to verify that I did make this information publicly accessible, which I always tell everyone else that they should do). 

Creating the website is therefore a useful tool for myself to keep track of what I'm doing. People on twitter have pointed out in their comments that it can also be useful for others: not only for the fictional Professor Awesome who is only waiting to make you a job offer, but also, for example, for students who would like to apply for a PhD at your department and are interested to get more information about what people in the department are doing.

I have included information about ongoing projects, published articles, and projects-on-hold. Including information about unpublished projects could be controversial: given that the preprints are presented alongside with published papers, unsuspecting readers might get confused and mistake an unpublished study for a peer-reviewed paper. However, I think that the benefits of making data and materials for unpublished studies outweighs the cost. Some of these papers are unpublished for practical reasons (e.g., because I ran out of resources to conduct a follow-up experiment). Even if an experiment turned out to be unpublishable because I made some mistakes in the experimental design, other people might learn from my mistakes in conducting their own research. This is one of the main reasons why I created the website: To make all aspects of all of my projects fully accessible.

Conclusion

As with everything, there are pros and cons with creating a personal website. A con is that some people might perceive you as narcissistic. There are many pros, though: Especially as an ECR, you provide a platform with information about your work which will remain independently of your employment status. You increase your visibility, so that others can contact you more easily. You can control what others can find out about you. And, finally, you can provide information about your work that, for whatever reason, does not come across in your publication list. So, in conclusion: I recommend to ECRs to make their own website.

Why I don’t wish that I had never learned about the replication crisis

Doing good science is really hard. It takes a lot of effort and time, and the latter is critical for early-career researchers who have a very limited amount of time to show their productivity to their next potential employer. Doing bad science is easier: It takes less time to plan an experiment, one needs less data, and if one doesn’t get good results immediately (one hardly ever does), the amount of time needed to make the results look good is still less than the amount of time needed to plan and run a well-designed and well-powered study.

Doing good science is frustrating at times. It can make you wonder if it’s even worth it. Wouldn’t life be easier if we were able to continue running our underpowered studies and publish them in masses, without having a bad conscience? I often wonder about this. But the grass always looks greener from the other side, so it’s worth taking a critical look at the BC (before-crisis) times before deciding whether the good old days really were that good.

I learned about the replication crisis gradually, starting in the second year of my PhD, and came to realise its relevance for my own work towards the end of my PhD. During my PhD, I conducted a number of psycholinguistic experiments. I knew about statistical power, in theory – it was that thingy that we were told we should calculate before we start an experiment, but nobody really does that, anyway. I collected as many participants as was the norm in my field. And then the fun started: I remember sleepless nights, followed by a 5-am trip to the lab because I’d thought of yet another analysis that I could try. Frustration when also that analysis didn’t work. Doubts about the data, the experiment, my own competence, and why was I even doing a PhD? Why was I unable to find a simple effect that others had found and published? Once, I was very close to calling the university which gave me my Bachelor of Science degree, and asking them to take it back: What kind of scientist can’t even replicate a stupid effect?

No, I don’t wish that I had never learned about the replication crisis. Doing good science is frustrating at times, but much more satisfying. I know where to start, even if it takes time to get to the stage when I have something to show. I can stand up for what I think is right, and sometimes, I even feel that I can make a difference in improving the system.