Crossroads of an early career researcher: Choosing what is important

         A while ago, I came across a table on Twitter ***** which lists the conflicts between being a good scientist versus being a good academic. It is shown below:

            I’m not sure if it was created as a satire or as genuine advise. It reads like satire, yet sadly, I see is no exaggeration when I compare it to my experience with academia so far.

            There seems to be a general trend towards improvement, where incentives in the academic system are increasingly based on the quality of research, not on its quantity, novelty, or sexiness. Just to name two examples: It was recently announced that hiring decisions at the psychology department of the LMU in Munich will now take into account the applicant’s dedication to open science; and, in my own experience, journal editors and reviewers have gotten stricter when it comes to papers with smallish sample sizes, which will eventually lead to an overall increase in statistical power and the informational value of studies. Generally, there is a lot of discussion about improving the reliability of scientific finding by supporting good science, and optimists point out that the incentives are changing, for the better, at an unprecedented rate.

However, at this stage, the change of incentives does not seem to have trickled down to the level of early career researchers (ECRs). This is unfortunate, because these changes might be especially important for ECRs*. Most of us have short-term contracts, and we are forced to decide whether we want to use our limited time being good scientists, or good academics. Despite the positive changes in the incentive structures, I face daily reminders of publication pressure. In a talk to a potential future employer, one of the first questions was about my publication record. A well-meaning and supportive senior colleague asked me, in a strict tone, whether I was publishing. In response to my cynical reply, “Well, I’m getting rejected a lot”, I got a lecture about the need to publish lots of papers, so I can compete with my peers for funding and jobs. To my knowledge, there are no grants or fellowships for ERCs which use indicators of good scientific practices rather than the number of papers to assign funding. This will certainly change if the upwards trend continues, but due to short-term contracts, the changes might not be implemented by the time that current post-docs or PhD students will need a job.

Given the current situation, I decided to take a step back, and think about what is important for me – in the long run. At least for now, it is clear that one cannot be a good scientist and a successful scientist at the same time. Being a good scientist (as defined in the table above) requires thorough planning of experiments, testing of possibly hundreds participants (in my area of research, participants are generally tested individually for 30 minutes), careful data analysis, follow-up experiments to clarify messy findings, etc., etc. In the time that it takes to publish one good study**, a peer can publish at least five sloppy studies.

It all comes down, then, to what is important to me as a person. I want to be successful in my career, of course, but I also want to be a good scientist. I chose this career – and, despite some ups and downs, I have never regretted this decision! – because I find it interesting to find and connect puzzle pieces that make up a bigger picture of how the world works. Engaging in practices that are currently required in order to be a successful scientist goes directly against this ideal. In other words, when I chose my career, I did not want to play a dirty game of selling myself, sucking up to the right people, and publishing results that I don’t even believe in myself. If I have a choice between being a good scientist and possibly having to leave academia once my post-doc contract runs out, and between engaging in practices that go against my principles and ideals, I choose the former. This is the conclusion that I came to, and I decided to write up a set of guidelines that I will follow, with the hope that I will be able to continue my career in science, which so far has been incredibly interesting and rewarding, while staying true to my ideals.

Maximise the evidential value of my research

Sloppily designed research wastes the time of participants, collaborators, reviewers, and myself. In the worst case scenario, a sloppy experiment may end up unpublishable if the results cannot be interpreted, or in the best case scenario it may be publishable with a bit of HARKing and p-hacking. It is not likely to yield good science. To maximise the chance of getting meaningful results, I will consider the following issues in planning, conducting, and publishing experiments:

-                    A careful consideration of the paradigm: Is the manipulation likely to work? Do a pilot test if unsure, and report it at such in any subsequent paper or pre-registration report.

-                    Power calculations: What effect size can I reasonably expect? Can I test enough participants to draw meaningful conclusions, regardless of the outcome of the experiment?

-                    Pre-register studies if they test a specific hypothesis, clearly label all exploratory analyses.

-                    Open materials, data, and analysis scripts: I have an account on the Open Science Framework, with all the materials, data and analyses for ongoing projects. This way, anyone can replicate my experiments or verify my analyses. If they find a mistake, it will allow for the correction of an erroneous result, and I will learn from it.

-                    Keep up to date with the literature on sound methodological and statistical methods, so I can make informed decisions in a study’s design and analysis.

-                    Never submit a paper, unless I can convince myself that the conclusions are justified while wearing my sceptic’s hat. Run past a critical colleague if unsure.

Interacting with others

-                    Be (healthily) sceptical about existing findings.

-                    Choose collaborators based on the quality of their research, and their attitude towards practices that support good rather than successful science (which goes hand in hand, I think).

-                    Sign reviews. I had a good experience with one of my first rejections: one of the reviews was negative, and the reviewer had signed. The reviewer had pointed out a potential confound, which pretty much destroyed the paper. Knowing his identity allowed me to contact him to further discuss this problem – which led to new, better experiments, and a deeper understanding of some theoretical issues on my part.

-                    When reviewing a paper, ask for data and materials. This is part of the Peer Reviewers’ Openness Initiative, of which I am a signatory.

Work-life balance

I added this section after having read a blog post by Jacob Jolij. Working long hours may increase the quantity of papers, but it doesn’t lead to better science – and it is bad for one’s health (as I have seen with friends and colleagues). As much as I enjoy research, it is also important for me to spend time with family and friends, and with my hobbies. Some resolutions to this end:

-                    Don’t goof off during working hours; no facebook or twitter. Turn off internet access (i.e., emails, the temptation to check facebook) for half of the day. If I do all the work during the day, I won’t feel guilty about not working in my free time.

-                    Rarely check email outside of working hours. Don’t check email at all during breakfast.

-                    Don’t work on weekends, unless I really, really want to. ***

Concluding remarks

            I should say that up to now, my experience in science has been predominantly positive****, and that I have been very fortunate in working together with people who encourage good science and integrity. This is probably why I am optimistic about the future: I hope that I can strive towards doing good science, and advance in my career conditional on the quality of my work.

            Finally, I should add that I was unsure whether I should publish this blog post. In the best case scenario, other ECRs might find it a useful reminder to sit down and decide what is important for them. In the worst case scenario, a potential future funder will read this, and decide that I am just trying to use some ideals to justify my low productivity. Hmm…. *shrugs*.

**********************************************

* Admittedly, I am not unbiased.

** Especially with limited financial resources.

*** It does happen sometimes.

**** Aside from my ongoing efforts to publish my 8 studies failing to replicate an effect.

**********************************************

***** Edit (10/3/16): I found the source of this table: it's this blog post by Gerald Carter: http://socialbat.org/2015/08/12/goals-of-science-vs-goals-of-scientists-a-love-letter-for-plos-one/, which is an interesting read about the role of alternative journals in improving the quality of science.