I followed my partner to a workshop in plasma physics. The
workshop was held in a mountain resort in Poland – getting there was an
adventure worthy, perhaps, of a separate blog post.
“I’m probably the only non-physicist in the room”, I say, apologetically,
at the welcome reception, when professors come up to me to introduce
themselves, upon seeing an unfamiliar face in the close-knit fusion
community.
Remembering this XKCD comic,
I ask my partner: “How long do you think I could pretend to be a physicist
for?” I clarify: “Let’s say, you are pretending to be a psychological
scientist. I ask you what you’re working on. What would you say?”
“I’d say: ‘I’m working on orthographic depth and how affects
reading processes, and also on statistical learning and its relationship to
reading’.” Pretty good, that’s what I would say as well.
“So, if you’re pretending to be a physicist, what would you
say if I ask you what you’re working on?”, he asks me.
“I’m, uh, trying to implement… what’s it called… controlled
fusion in real time.”
The look on my partner’s face tells me that I would not do
very well as an undercover agent in the physics community.
The attendees are around 50 plasma physicists, mostly
greying, about three women among the senior scientist, perhaps five female
post-docs or PhD students. Halfway through the reception dinner, I am asked
about my work. In ten sentences, I try to describe what a cognitive
scientist/psycholinguist does, trying to make it sound as scientific and
non-trivial as possible. Several heads turn, curious to listen to my
explanation. I’m asked if I use neuroimaging techniques. No, I don’t, but a lot
of my colleagues and friends do. For the questions I’m interested in, anyway, I
think we know too little about the relationship between brain and mind to make
meaningful conclusions.
“It’s interesting”, says one physicist, “that you could
explain to us what you are doing in ten sentences. For us, it’s much more
difficult.” More people join in, admitting that they have given up trying to
explain to their families what it is they are doing.
“Ondra gave me a pretty good explanation of what he is
doing”, I tell them, pointing at my partner. I sense some scepticism.
Physics envy is a term coined by psychologists (who else?),
describing the inferiority complex associated with striving to be taken serious
as a field in science. Physics is the prototypical hard science: they have long
formulae, exact measurements where even the fifth decimal places matter, shiny multi-billion-dollar
machines, and stereotypical crazy geniuses who would probably forget their own
head if it wasn’t attached to them. Physicists don’t
always make it easy for their scientific siblings (or distant cousins)* but, admittedly, they do have a right to be smug towards psychological scientists, given the
replication crisis that we’re going through. The average physicist,
unsurprisingly, finds it easier to grasp concepts associated with mathematics
than the average psychologist. This means that physicist have, in general, a better
understanding of probability. When I tell physicists about some of the absurd
statements that some psychologists have made (“Including unpublished studies in
the meta-analysis erroneously biases an effect size estimate towards zero.”;
“Those replicators were just too incompetent to replicate our results. It’s
very difficult to create the exact conditions under which we get the effect:
even we had to try it ten times before we got this significant result!”),
physicists start literally rolling on the floor with laughter. “Why do you even
want to stay in this area of research?” I was asked once, after the physicist I
was talking to had wiped off the tears of laughter. The question sounded
neither rhetorical nor snarky, so I gave a genuine answer: “Because there are a
lot of interesting questions that can be answered, if we improve the
methodology and statistics we use.”
In physics, I am told, no experiment is taken seriously
until it has been replicated by an independent lab. (Unless it requires some unique equipment, in which case it can't be replicated by an independent lab.) Negative results are still considered informative, unless they are due to experimental errors. Physicists still have issues with
researchers who make their results look better than they actually are by
cherry-picking the experimental results that fit best within one’s hypothesis
and with post-hoc parameter adjustments – after all, the publish-or-perish
system looms over all of academia. However, the importance of replicating
results is a lesson that physicists have learnt from their own replication crisis:
in the late 1980s, there was a
shitstorm about cold
fusion, set off by experimental results that were of immense
public interest, but theoretically implausible, difficult to replicate, and
later turned out to be due to sloppy research and/or scientific
misconduct. (Sounds familiar?)
Physicists take their research very seriously, probably to a
large extent because it is often of great financial interest. There are those
physicists who work closely with industry. Even for those who don’t, their work
often involves very expensive experiments. In plasma physics, a shot on the
machine of Max Planck Institute for Plasma Physics, ASDEX-Upgrade, costs several
thousand dollars. The number of shots required for an experiment depends on the research aims, and whether there is other data available, but can go up to 50 or more. This gives very strong
motivation to make sure that one’s experiment is based on accurate calculations
and sound theories which are supported by replicable studies. Furthermore, as
there is only one machine – and only a handful of similar machines all over
Europe – it needs to be shared with all other internal and external projects.
In order to ensure that shots (and experimental time) are not wasted, any team
wishing to perform an experiment needs to submit an application; the call for
proposals opens only once a year. A representative of the team will also need
to do a talk in front of the committee, which consists of the world’s leading
experts in the area. The committee will decide whether the experiment is likely
to yield informative and important results. In short, it is not possible – as
in psychology – to spend one’s research career testing ideas one has on a whim,
with twenty participants, and publish only if it actually ‘works’. One would be
booed of the stage pretty quickly.
It’s easy to get into an us-and-them mentality and feelings
of superiority and inferiority. No doubt all sciences have something of
importance and of interest to offer to society in general. But it is also
important to understand how we can maximise the utility of the research that we
produce, and in this sense we can take a leaf out of physicists’ books. The
importance of replication should be adopted also into the psychological
literature: arguably, we should simply forget all theories that are based on
non-replicable experiments. Perhaps more importantly, though, we should start
taking our experiments more seriously. We need to increase our sample sizes;
this conclusion seems to be gradually coming through as a consensus in
psychological science. This means that also our experiments will become more
expensive, both in terms of money and time. By conducting sloppy studies, we
may still not loose thousands of dollars of taxpayers’ (or, even worse,
investors’) money for each blotched experiment, but we will waste the time of
our participants, the time, nerves and resources of researchers who try to make
sense of or replicate our experiments, and we stall progress in our area of
research, which has strong implications for policy makers in areas ranging from
education through improving social equality, prisoners’ rehabilitation, and political/financial
decision making, to mental health care.
--------------------------------------
* Seriously, though, I haven’t met a physicist who is as bad
as the linked comic suggests.
Acknowledgement: I'd like to thank Ondřej Kudláček, not only for his input into this blogpost and discussions about good science, but also for his unconditional support in my quest to learn about statistics.