In our reading group, we discussed a landmark paper of Paul
Meehl’s, “Why summaries of research on psychological theories are often
unintepretable” (1990). The paper ends with a very strong statement (p. 242),
written by Meehl in italics for extra emphasis:
We should maturely and
sophisticatedly accept the fact that some perfectly legitimate “empirical”
scientific theories may not be strongly testable at a given time, and that it
is neither good scientific strategy nor a legitimate use of the taxpayer’s
dollar to pretend otherwise.
This statement should bring up all
kinds of stages of grief in psychological researchers, including anger, denial,
guilt, and depression. Are we really just wasting taxpayers’ money on studying
things that are not studyable (yet)?
We sometimes have ideas, theories, or models, which cannot
be tested given our current measurement devices. However, research is a process
of incremental progress, and in order to make progress, we need to first
understand if something works or not, and if not, why it doesn’t work. If we
close our eyes towards all of the things that don’t work, we cannot progress.
Even worse, if we find out that something doesn’t work, and don’t make any
effort to publicise our results, other researchers are likely to get the same idea, at some point in time, and start using their resources in order to also find out that it doesn’t work.
To illustrate with a short example: For some reason or
another, I decided to look at individual differences in the size of
psycholinguistic marker effects. With the help of half a dozen colleagues, we
have collected data from approximately 100 participants, tested individually in
1-hour sessions. The results so far suggest that this approach doesn’t work:
there are no individual differences in psycholinguistic marker effects.
Was I the first one to find this out? Apparently not. When
sharing my conclusion with some older colleagues, they said: “Well, I could
have told you that. I have tried to
use this approach for many years with the same results.” Could I have known
this? Did I waste the time of my colleagues and the participants in pursuing something
that everyone already knows? I think not. At least myself and my colleagues
were unaware of any potential problems with this approach. And finding out that
it doesn’t work opens interesting new questions: Why doesn’t it work? Does it
work in some other populations? Can we make it work?
All of these questions are important, even if the answer is
that there is no hope to make this approach work. However, in the current
academic reward system, studying things that may never work is not a good strategy. If one
wants publications, a better strategy is to drop a study like a hot potato once
you realise that it will not give a significant result: throw it
into your file drawer and move on to something else, something that will be
more likely to give you a significant p-value
somewhere. This is waste of taxpayer’s
money.
