I’m not
sure if it was created as a satire or as genuine advise. It reads like satire,
yet sadly, I see is no exaggeration when I compare it to my experience with
academia so far.
There seems
to be a general trend towards improvement, where incentives in the academic
system are increasingly based on the quality of research, not on its quantity,
novelty, or sexiness. Just to name two examples: It was recently announced that
hiring decisions at the psychology
department of the LMU in Munich will now take into account the applicant’s
dedication to open science; and, in my own experience, journal editors and
reviewers have gotten stricter when it comes to papers with smallish sample
sizes, which will eventually lead to an overall increase in statistical power
and the informational value of studies. Generally, there is a lot of discussion
about improving the reliability of scientific finding by supporting good
science, and optimists point out that the incentives are changing, for the
better, at an unprecedented rate.
However, at this stage, the
change of incentives does not seem to have trickled down to the level of early
career researchers (ECRs). This is unfortunate, because these changes might be
especially important for ECRs*. Most of us have short-term contracts, and we
are forced to decide whether we want to use our limited time being good
scientists, or good academics. Despite the positive changes in the incentive
structures, I face daily reminders of publication pressure. In a talk to a potential
future employer, one of the first questions was about my publication record. A
well-meaning and supportive senior colleague asked me, in a strict tone,
whether I was publishing. In response to my cynical reply, “Well, I’m getting
rejected a lot”, I got a lecture about the need to publish lots of papers, so I
can compete with my peers for funding and jobs. To my knowledge, there are no
grants or fellowships for ERCs which use indicators of good scientific
practices rather than the number of papers to assign funding. This will
certainly change if the upwards trend continues, but due to short-term
contracts, the changes might not be implemented by the time that current
post-docs or PhD students will need a job.
Given the current situation, I
decided to take a step back, and think about what is important for me – in the
long run. At least for now, it is clear that one cannot be a good scientist and
a successful scientist at the same time. Being a good scientist (as defined in
the table above) requires thorough planning of experiments, testing of possibly
hundreds participants (in my area of research, participants are generally
tested individually for 30 minutes), careful data analysis, follow-up
experiments to clarify messy findings, etc., etc. In the time that it takes to
publish one good study**, a peer can publish at least five sloppy studies.
It all comes down, then, to what
is important to me as a person. I want to be successful in my career, of
course, but I also want to be a good scientist. I chose this career – and,
despite some ups and downs, I have never regretted this decision! – because I
find it interesting to find and connect puzzle pieces that make up a bigger
picture of how the world works. Engaging in practices that are currently
required in order to be a successful scientist goes directly against this
ideal. In other words, when I chose my career, I did not want to play a dirty
game of selling myself, sucking up to the right people, and publishing results
that I don’t even believe in myself. If I have a choice between being a good
scientist and possibly having to leave academia once my post-doc contract runs
out, and between engaging in practices that go against my principles and
ideals, I choose the former. This is the conclusion that I came to, and I
decided to write up a set of guidelines that I will follow, with the hope that
I will be able to continue my career in science, which so far has been
incredibly interesting and rewarding, while staying true to my ideals.
Maximise the evidential value of my research
Sloppily designed research wastes
the time of participants, collaborators, reviewers, and myself. In the worst
case scenario, a sloppy experiment may end up unpublishable if the results
cannot be interpreted, or in the best case scenario it may be publishable with
a bit of HARKing
and p-hacking. It is not likely to yield good science. To maximise
the chance of getting meaningful results, I will consider the following issues
in planning, conducting, and publishing experiments:
-
A careful consideration of the paradigm: Is the manipulation
likely to work? Do a pilot test if unsure, and report it at such in any
subsequent paper or pre-registration report.
-
Power calculations: What effect size can I
reasonably expect? Can I test enough participants to draw meaningful
conclusions, regardless of the outcome of the experiment?
-
Pre-register studies if they test a specific
hypothesis, clearly label all exploratory analyses.
-
Open materials, data, and analysis scripts: I
have an account on the Open Science Framework, with all
the materials, data and analyses for ongoing projects. This way, anyone can
replicate my experiments or verify my analyses. If they find a mistake, it will
allow for the correction of an erroneous result, and I will learn from it.
-
Keep up to date with the literature on sound
methodological and statistical methods, so I can make informed decisions in a study’s design and analysis.
-
Never submit a paper, unless I can convince
myself that the conclusions are justified while wearing my sceptic’s hat. Run
past a critical colleague if unsure.
Interacting with others
-
Be (healthily) sceptical about existing
findings.
-
Choose collaborators based on the quality of
their research, and their attitude towards practices that support good rather
than successful science (which goes hand in hand, I think).
-
Sign reviews. I had a good experience with one
of my first rejections: one of the reviews was negative, and the reviewer had
signed. The reviewer had pointed out a potential confound, which pretty much
destroyed the paper. Knowing his identity allowed me to contact him to further
discuss this problem – which led to new, better experiments, and a deeper
understanding of some theoretical issues on my part.
-
When reviewing a paper, ask for data and
materials. This is part of the Peer Reviewers’ Openness
Initiative, of which I am a signatory.
Work-life balance
I added this section after having read a blog post by Jacob Jolij. Working long
hours may increase the quantity of papers, but it doesn’t lead to better
science – and it is bad for one’s health (as I have seen with friends and
colleagues). As much as I enjoy research, it is also important for me to spend
time with family and friends, and with my hobbies. Some resolutions to this
end:
-
Don’t goof off during working hours; no facebook
or twitter. Turn off internet access (i.e., emails, the temptation to check
facebook) for half of the day. If I do all the work during the day, I won’t
feel guilty about not working in my free time.
-
Rarely check email outside of working hours.
Don’t check email at all during breakfast.
-
Don’t work on weekends, unless I really, really
want to. ***
Concluding remarks
I should
say that up to now, my experience in science has been predominantly
positive****, and that I have been very fortunate in working together with people who encourage
good science and integrity. This is probably why I am optimistic about the
future: I hope that I can strive towards doing good science, and advance in my
career conditional on the quality of my work.
Finally, I
should add that I was unsure whether I should publish this blog post. In the
best case scenario, other ECRs might find it a useful reminder to sit down and
decide what is important for them. In the worst case scenario, a potential
future funder will read this, and decide that I am just trying to use some
ideals to justify my low productivity. Hmm…. *shrugs*.
**********************************************
* Admittedly, I am not unbiased.
** Especially with limited financial resources.
*** It does happen sometimes.
**** Aside from my ongoing efforts to publish my 8 studies
failing to replicate an effect.
**********************************************
***** Edit (10/3/16): I found the source of this table: it's this blog post by Gerald Carter: http://socialbat.org/2015/08/12/goals-of-science-vs-goals-of-scientists-a-love-letter-for-plos-one/, which is an interesting read about the role of alternative journals in improving the quality of science.
**********************************************
***** Edit (10/3/16): I found the source of this table: it's this blog post by Gerald Carter: http://socialbat.org/2015/08/12/goals-of-science-vs-goals-of-scientists-a-love-letter-for-plos-one/, which is an interesting read about the role of alternative journals in improving the quality of science.
