Selective blindness to null results

A while ago, to pass time on a rainy Saturday afternoon, I decided to try out some publication bias detection techniques. I picked the question of gender differences in multitasking. After all, could there be a better question for this purpose than this universally known ‘fact’? I was surprised, however, to find not two, not one, but zero peer-reviewed studies that found that women were better at multitasking than men. The next surprise came when I started sharing my discovery with friends and colleagues. In response to my “Did you know that this women-are-better-at-multitasking-thing is a myth?” I would start getting detailed explanations about possible causes of a gender difference in multitasking.

Here is another anecdote: last year, I did a conference talk where I presented a null-result. A quick explanation of the experiment: a common technique in visual word recognition research is masked priming, where participants are asked to respond to a target word, which is preceded by a very briefly presented prime. The duration of the prime is such that participants don’t consciously perceive it, but the degree and type of overlap between the prime and the target affects the response times to the target. For example, you can swap the order of letters in the prime (jugde – JUDGE), or substitute them for unrelated letters (julme – JUDGE). I wanted to see if it matters whether the transposed letters in the prime create a letter pair that does not exist in the orthography. As it turns out, it doesn’t. But despite my having presented a clear null result (with Bayes factors), several people came up to me after my talk, and asked me if I thought this effect may be a confounding variable for existing studies using this paradigm!

Though I picked only two examples, such selective blindness (or deafness) to being told that an effect is not there seems to be prevalent in academia. I’m not just talking about instances of papers citing those articles which support their hypothesis, and conveniently forgetting that a handful of studies failed to find evidence for it (or citing them as providing evidence for it even when they don’t). In this case, my guess would be that there are numerous factors at play, including confirmation bias and deliberate strategies. In addition to this, however, we seem to have some mechanism to preferentially perceive positive results over null-results. This seems to go beyond the common knowledge that non-significant p-values cannot be interpreted as evidence for the null, or the (in many cases well-justified) argument that a null-result may simply reflect low power or incorrect auxiliary hypotheses. The lower-level blindness that I’m talking about could reflect our expectations: surely, if someone writes a paper or does a conference presentation, they will have some positive results to report? Or perhaps we are naturally tuned to understand the concept of something being there more readily than the concept of something not being there.

I’ve argued previously that we should take null results more seriously. It does happen that null results are uninterpretable or uninformative, but a strong bias towards positive results at any stage of the scientific discourse will provide a skewed view of the world. If selective blindness to null results exists, we should become aware of it: we can only evaluate the evidence if we have a full picture of it.